• Processing (Info)

    • MPC Status Page: Archive (2008 July-December)

    This page describes enhancements to or problems that have occurred with the MPC and scripts and the fixes that have been made.

    Recent problems are listed elsewhere. Index of other older problems..

    Older Enhancements and Resolved Problems

    • Missing Objects in MPChecker
      2008 Dec. 30: 17:50. A number of users have reported inconsistent output from MPChecker when checking more than one object. It seems that some code, recently added to remove duplicate objects from the output list, is removing objects duplicated in different fields. A fix is being worked on.
      • 19:00. A fix is now in place.

    • No NEO Observations on DOU MPECs
      2008 Dec. 23: 11:30. Recent DOU MPECs have not contained any new new NEO observations due to a problem with the routine that extracts the observations. A fix is in place. The missing observations should appear on the next DOU MPEC.

    • Erroneous Designations in MPChecker
      2008 Dec. 21: 10:00. A user reported that certain objects in MPChecker are not correctly labeled, specifically that unnumbered objects are labeled erroneously with numbered designations. The positions and motions are correct, but the designation is wrong. The problem seems to be restricted to numbered objects above (199999). We are investigating the problem.
      • 12:00. As suspected, the problem lay with the labeling of orbits for numbered objects above (199999). The problem is now fixed.

    • Issue with Missing Orbits During MPC Preparation
      2008 December 11: 11:45. There has been a long-standing problem with elements for newly-numbered objects being unavailable in the MPES for a period during MPC preparation. This has been a troublesome problem to investigate as the problem only occurs while we are busy with the filing of the new data, meaning we do not have time to examine the issue. We have now found a few free moments, have identified a possible source for the problem and put a fix in place. We will see if the problem disappears during the next MPC preparation period.
      • 15:15. Unfortunately, it appears that the above fix is causing severe performance problems for the MPES. An alternate fix will have to be found. The MPES executable has been restored to its previous version.

    • E-mailing of circulars
      2008 December 3: 10:25. The e-mailing of circulars issued since noon yesterday has been interrupted by problems caused by transfer of the mailing scripts from a Solaris machine that has been retired to a Linux box. The problems are being examined and a fix should be forthcoming shortly. The backed-up circulars will then start being delivered automatically.
      • 10:53. We believe that the problem (related to interpretation of relative pathnames between Solaris and Linux) has been fixed. The backed-up circulars are being delivered.

    • Anon-ftp disk full
      2008 November 27: 11:18. The disk on which the anon ftp files (such as the MPCORB datafiles) reside filled up overnight.
      • November 28: 09:15. The anon-ftp disk again filled up overnight.
      • December 2: 09:15. Someone has cleared space on the anon-ftp disk as it is currently 70 percent full.

    • Mid-month observation files in ECS
      2008 November 24: 11:00. A number of subscribers have reported that they cannot access the files in /ECS/MPCAT-OBS/midmonth/. It seems that this is related to the CF machine changeover mentioned below. We are investigating, but we have no information of what actually got executed on the Linux box.
      • 12:15. Some manual digging shows that a cron job script (to build the zipped versions of files in the /midmonth directory) did not run. This script also clears the lock that prevents users accessing the directory while the zipping is being performed. The script is being run manually. The cron job problem remains under investigation.
      • 13:05. We believe the problem is fixed.

    • SMTP E-mail Interruption (November 24)
      2008 November 19: 17:02. We have been informed that the CF's SMTP server will be off-line for approximately 30 minutes beginning at 06:05 EST on Monday, November 24, to allow OS patches to be installed. Attempts to send e-mail to us during this period will fail, but the mail should queue up on the sending computer for a subsequent resend attempt.

    • ECS files for November MPC
      2008 November 17: 12:50. The ECS files for the November MPCs are currently being posted. A large number of problems related to the Solaris-Linux transition of the CF machine (see earlier notes) have arisen. Users are warned to expect problems...

    • DISCSTATUS reports
      2008 November 15: 20:57. The naming status of numbered minor planets with numbers above 200000 is incorrect in the just-mailed DISCSTATUS reports. The count of oppositions may also be incorrect on some objects. The problem has been fixed and the next run will correctly flag these objects.

    • ECS files for November MPC
      2008 November 15: 10:10. Due to a rather knotty problem connected to the preparation of the PDFs of the three sets of circulars, the ECS files for the November MPCs will not be posted until late on Monday, November 17.

    • Anon-ftp disk full
      2008 November 15: 00:40. The disk on which the anon ftp files reside has again filled up. We have reported the problem to the CF.

    • Unavailability of CF Webserver
      2008 November 7: 20:00. The CF webserver is not serving up pages. The problem has been reported to the CF. Our cgi scripts continue to be accessible through the mirror page.
      • November 9: 00:39. We've just noticed that the CF webserver appears to be operational again. There was no notification from the CF.

    • Stale Information in MPES
      2008 November 6: 00:45. We have noticed that some information in the MPES is stale. The affected information is that derived from external sites and which is supposed to be copied from those sites on a regular basis via a number of Python scripts. Examples of such data files include lists of Arecibo and Goldstone radar targets. In some cases, the MPES is working from data files derived from files downloaded in September. This problem predates the Oct. 16 problem with a CF server, but the problem is OS related: the Python exectuable was moved from /opt/bin to /usr/bin. We believe all affected scripts have been modified and data files should become current after the next run of the relevant script.

    • E-mail Delivery of MPECs
      2008 October 30: 11:30. We have received a number of reports from subscribers who have not received recent MPECs. We have received our copies of recent circulars, so the problem would seem to be restricted to some/all external addresses. The machine that sends out the circulars is a CF machine. We are investigating...
      • 16:40. We are resending MPECs U92-U100.
      • 16:55. The first attempt used malformed scripts. We are trying again... Note that you will receive the resent DOU MPECs even if you've requested non-delivery of these circulars. We apologize for this minor inconvenience.

    • Network Interruptions (November 4)
      2008 October 29: 12:40. We have been informed that the Computation Facility will be performing software upgrades on core network equipment on Tuesday, November 4, from 06:00 to 07:30 EDT. We expect MPC (and other CfA) webpages to be unavailable during this period. Access via the mirror pages will also be affected.

    • MPCORB files and mirrors
      2008 October 20: 12.56. The updating of the MPCORB datafiles on our site and the mirrors has been affected by two related problems. The first is the previously-mentioned OS change on a CF machine. The second is the failure to copy files to the mirror due to a change in the host key. We are working on transferring new host keys to the remote site.

    • Latest mid-month MPS batch
      2008 October 19: 11.15. Last night's mid-month MPS batch failed to update correctly due to problems with the new CF machine mentioned previously. We will have to manually reconstruct the bits that failed, while modifying the scripts so that they work next time.
      • 18:15. We believe all the failed bits have been fixed.
      • October 20: 11.41 A user reported that the midmonth/mpu.* files are missing. That bit of the update procedure is being rerun.
      • 12:15. The midmonth/mpu.* files have been restored.

    • Network Interruptions (October 21)
      2008 October 16: 18:30. We have been informed that Harvard will be upgrading core network equipment on Tuesday, October 21, from 06:30 to 07:30 EDT. We expect MPC (and other CfA) webpages to be unavailable during this period. Access via the mirror pages will also be affected.

    • Power outage at Observatory II
      2008 October 16: 12:00. Yesterday's power outage fried one of the CF servers. Unfortunately, it was the machine on which many of the cron jobs run that handle data transfer between the MPC and the CF webserver. A new machine is in place, but the cron jobs will not be restored until early next week. We will try and replace as much as we can, but some web pages and data files may not get updated as expected.
      • October 18: 12:00. Some of the cron jobs are working again, but others continue to cause problems. This is a result of the old machine being a Solaris box while the new machine is a Linux box. Specifically, the preparation of the CR-terminated versions of various MPCORB datafiles is failing due a segmentation fault. Also, the copying of files to the official mirror site is failing due an incompatability in specifying batch mode in the command-line flags: the original version of sftp used -b to specify batch mode; an "upgrade" to sftp caused batch mode to be specified with -B; the version on the new Linux machine again uses -b! We believe that all cron jobs that use sftp have been modified to reflect the flag usage de jour.
      • 18:00. The October Minor Planet Circulars will not be published, partly due to the power outage and partly due to staff absences.

    • Power outage at Observatory
      2008 October 15: 12:51. An unexpected power outage affected the Observatory (and, apparently, parts of Cambridge) around 10:50 today. The MPC machines began shutting off 15 to 20 minutes later when they exceeded the capacity of the UPSes. The power has just been restored, the cluster has reformed and processing routines are being restarted. Normalcy should be resumed shortly.
      • 13:15. While the MPC's systems are fully functional at this point, it seems that (at least some parts of) the Computation Facilities are not. We are not receiving e-mail at this point and we cannot access CF-hosted machines. We expect these problems to disappear shortly.
      • 14:20. E-mail from outside is beginning to arrive.
      • 23:55. There were problems with the MPC webserver. The webserver was restarted.

    • MPEC 2008-T101
      2008 October 8: 17:15. Due to a production error (two people working on announcing a new object at the same time), the object 2008 TK9, which appeared on MPEC 2008-T100, was also announced as 2008 TL9 on MPEC 2008-T101. The latter designation and MPEC have been retracted. The next MPEC to be issued will be 2008-T101.

    • Linking of designations in MPES
      2008 October 8: 01:25. A user reported that entering the non-principal designation 2008 SC7 into the MPES did not return the elements or ephemeris for the principal designation 2004 XX145. The problem, which presumably affected other objects as well, was traced to a malformed entry in the new double designation file and fixed.

    • No space left on on ftp server
      2008 September 28: 15:00. The anon ftp server disk is again full (currently only zero bytes are free). Updating of various files stored there may fail until such time as space is available.

    • No space left on on ftp server
      2008 September 19: 14:27. The anon ftp server disk is again full (currently only 4 KB are free). Updating of various files stored there may fail until such time as space is available.

    • Corruption of files on ftp server
      2008 August 28: 11:13. It has been reported that the MPCORBcr.DAT file is corrupted. Investigation shows that a number of files in the pub/MPCORB directory are blank or incomplete. There is an error message in the logfile of last night's DOU MPEC preparation, but it isn't very helpful.
      • August 29: 11:08. The problem is still present after last night's DOU MPEC. The same cryptic, unhelpful error message is present in the DOU preparation logfile.
      • 11:17. It seems that the disk on which the anon FTP data sits is full.
      • 14:00. We have cleared over 1 GB of no-longer-needed material from the anon-ftp disk.
      • 21:30. As of now, there are only 8 KB free on the anon FTP disk. Tonight's update is therefore likely to fail and since it's a long holiday weekend in the U.S. space may not get freed up until the middle of next week. Unfortunately, this is a disk that we do not control.
      • 22:10. Some other user has apparently cleared up old files, as there is now 1.1 GB free. Of course, this free space may disappear before the DOU MPEC procedure runs...

    • Designation Message Problem
      2008 July 28: 12:50. It has been noted that some observers are receiving some incorrect correspondences between their temporary designations and MPC designations. These designations are coming from a reprocessing of the old unidentified material. The problem occurred because the reprocessing is being handled in a different mode than the normal processing. Further, this problem affects only the messages sent to the observers. A fix to avoid this confusing problem has been implemented.

    • SMTP E-mail Interruption (August 4)
      2008 July 22: 16:45. We have been informed that the CF's SMTP server will be off-line for approximately 60 minutes beginning at 06:00 EDT on Monday, August 4, to allow OS patches to be installed. Attempts to send e-mail to us during this period will fail, but the mail should queue up on the sending computer for a subsequent resend attempt.

    • CF Webserver Outage
      2008 July 8: 23:33. The Computation Facility's webserver has apparently died as attempts to access any of our pages (or the SAO main page) all time out. The problem was reported to the CF at 23:32:12 EDT. Note that access to various observer services via the mirror pages was unaffected by this problem.
      • July 9: 09:00. The CF reports that the webserver has been restarted.

    • SMTP E-mail Interruption (July 7)
      2008 July 3: 13:13. We have been informed that the CF's SMTP server will be off-line for approximately 30 minutes beginning at 06:00 EDT on Monday, July 7, to allow OS patches to be installed. Attempts to send e-mail to us during this period will fail, but the mail should queue up on the sending computer for a subsequent resend attempt.