• Processing (Info)

    • MPC Status Page: Archive (2008 January-June)

    This page describes enhancements to or problems that have occurred with the MPC and scripts and the fixes that have been made.

    Recent problems are listed elsewhere. Index of other older problems..

    Older Enhancements and Resolved Problems

    • Network Interruptions (June 30)
      2008 June 30: 17:00. We have been informed (at very short notice) that the border firewall at CfA will be undergoing service between 18:00 and 20:00 EDT tonight. Connections from/to the outside world to CfA will be affected by this outage. Internal connectivity should not be affected.

    • Network Interruptions (June 29)
      2008 June 18: 17:00. We have been informed that the Computation Facility will be upgrading core network equipment on Sunday, June 29, from 13:00 to 22:00 EDT. We expect MPC (and other CfA) webpages to be unavailable during this period. Access via the mirror pages will also be affected.

    • Unavailability of MPC webpages (June 23)
      2008 June 17: 16:58. The MPC webpages hosted in the Computation Facility (as well as all CfA webpages so hosted) will be unavailable between 17:45 EDT and (approximately) 19:00 EDT on Monday, June 23, as a result of further disk upgrades being undertaken by the CF. Access to MPC cgi scripts via the mirror pages should be unaffected by this upgrade.

    • Spam filtering at CfA
      2008 June 4: 23:30. The Computation Facility has recently activated a new spam filtering system at the CfA. All e-mail addresses at cfa.harvard.edu are subject to this filtering system which is administered by an outside service. So far, two observation batches have been incorrectly flagged as spam and quarantined. Both batches have since been released for delivery, but until the filtering service learns that observation batches are not spam it may be the case that certain batches do not get delivered to the MPC in a timely fashion.

    • Unavailability of MPC webpages
      2008 May 29: 18:41. The MPC webpages hosted in the Computation Facility (as well as all CfA webpages so hosted) seem to be unavailable for reason(s) not yet explained by the CF.
      • 21:45. The problem, whatever it was, seems to have been fixed.

    • Unavailability of MPC webpages
      2008 May 28: 10:41. The MPC webpages hosted in the Computation Facility (as well as all CfA webpages so hosted) will be unavailable between 17:15 EDT and (approximately) 21:00 EDT as a result of further disk upgrades being undertaken by the CF. Access to MPC cgi scripts via the mirror pages should be unaffected by this upgrade.

    • Unavailability of MPC webpages
      2008 May 20: 10:41. The MPC webpages hosted in the Computation Facility (as well as all CfA webpages so hosted) will be unavailable between 17:15 EDT and (approximately) 20:00 EDT as a result of disk upgrades being undertaken by the CF. Access to MPC cgi scripts via the mirror pages should be unaffected by this upgrade.

    • Upcoming Network Maintenance (May 29)
      2008 May 19: 10:11. We have been informed that Harvard will be performing maintenance on one of its gateway machines on Thursday, May 29. The work will begin at 05:00 EDT and should be completed by 07:00. During this maintenance window there will may be intermittent connectivity into and out of the observatory.

    • Unavailability of MPC webpages
      2008 May 14: 15:35. The MPC webpages hosted in the Computation Facility (as well as all CfA webpages so hosted) will be unavailable for about 10 minutes starting at 18:15 today due to scheduled maintenance.

    • Unavailability of MPC webpages
      2008 May 5: 18:30. The MPC webpages hosted in the Computation Facility (as well as all CfA webpages so hosted) were unavailable beginning at 17:30 due to disk upgrades. This work was announced but we missed the fact that it would impact webpage availability. The work has been completed.

    • Upcoming Network Maintenance (April 24)
      2008 Apr. 17: 11:01. We have been informed that Harvard will be performing maintenance on one of its gateway machines on Thursday, April 24. The work will begin at 05:30 EDT and should be completed by 07:00. During this maintenance window there will may be intermittent connectivity into and out of the observatory.

    • E-mail problems
      2008 Apr. 11: 15:00. As a result of a recent unannounced change in the e-mail system at CfA, there has been some disruption to our internal mailings over the past few days. There may also have been some disruption to outgoing e-mail. We have made the necessary changes to our e-mail setup and believe that all is back to normal.

    • NEO Observations on MPEC 2008-G46
      2008 Apr. 9: 11:00. As a result of a designation snafu yesterday, some observations attributed to objects with K08G00x designations do not belong. These observations will be appearing on forthcoming MPECs under their intended designations.

    • Mailing of Circulars
      2008 Mar. 29: 17:10. The machine used to mail out circulars (MPECs, IAUCs and CBETs), as well as Discovery Status reports, has apparently died. This machine is administered by the Computation Facility. They have been informed of the problem, but it is unlikely that they will look at the problem until Monday and if the machine has died it may be some time before a replacement is available. We will have to setup an alternate method of mailing out circulars.
      • 21:24. A partially-automated method of mailing out the circulars has been implemented. DOU MPECs will be delayed in their e-mailing.
        Subscribers who check sender addresses are warned that e-mails will not appear to come from the standard sources. Checks for e-mails should allow any address @cfa.harvard.edu. Failure to do modify the checks as mentioned here will cause you to miss circulars.
      • Mar. 31: 16:00. The immediate problem with the machine has been resolved. Automatic mailing of circulars has resumed.

    • Webserver problem
      2008 Mar. 8: 12:00. A user has reported that "file locked" errors are still occurring when accessing web services. The AUTOACK problems, combined with staff absences, mean that it is unlikely that this problem can be looked at until next week.
      • Mar. 14. 12:00. Examination of the configuration of the new webserver showed one difference when compared to that of the old webserver. This change was made in the configuration and the webserver was restarted. We believe this will solve the problem: please let us know if this isn't the case!

    • AUTOACK problem
      2008 Mar. 6: 17:20. A problem with DNS within our local network is causing mail sent to mpc@cfa to bounce back from the machine that processes incoming e-mail, because it thinks the message is to be relayed to another machine (which is forbidden). We are trying to get this fixed as soon as possible. This is something that we are dependent on our Computation Facility to fix.
      • 19:07. Not fixed as of now. Observers should continue to send observations as normal. Observation batches will be forwarded manually to the processing queues by MPC staff. Observers should ensure that every batch of observations has the AC2 keyword, otherwise they may not receive an ACK. Expect delays in ACKs overnight, as there will be no manual submitting between 23:30 and 09:30.
      • Mar. 7: 19:50. The DNS problem is still not fixed. It is unlikely that it will now be fixed before Monday. Expect delays in ACKs until at least Monday, with associated delays in returning designations to observers. Observers are asked to refrain from resending observations if they get bouncebacks containing: 
                ----- The following addresses had permanent fatal errors -----
     autoack@ubasti
         (reason: 551 User not local, Relay disabled.)
         (expanded from: <MPC@cfa.harvard.edu>)
        If you get this error, we have your observations, you will be sent an ACK (albeit somewhat delayed) and the observations will be processed.
      • Mar. 8: 23:15. Since staff members also get copies of e-mails sent to mpc@cfa, I have just finished coding a routine to forward observation e-mails from my personal e-mail account to the automated routines. This routine is being run every 97 seconds, although it will not run between 04:00 and (possibly as late as) 05:00 EDT. As noted previously, you must use the AC2 keyword in order to receive ACKs. It is also worth remarking that the forwarding introduces some additional material into the message which will raise your "junk rating" score.
      • Mar. 10: 18:00. Still no resolution in sight.
      • Mar. 11: 00:00. It seems that the problem was fixed an hour or more ago, but we've had no confirmation from the Computation Facility. The special routine mentioned in the Mar. 8 report has been stopped. Normal ACK'ing should occur from now on.
      • Mar. 11: 16.00. The fix seems stable. It may be a day or so before internal processing returns to normal: priority will be given to processing observations.

    • MPCORB.DAT file
      2008 Mar. 5: 09:00. The secondary datafile from which the MPCORB.DAT (and its varients) are produced was blanked out by last night's DOU MPEC process. Log files are being examined to see why this occurred. Once the reasons are understood, the file will have to be rebuilt.
      • Mar. 5: 11:15. The secondary datafile has been rebuilt. The publically-accessible MPCORB file will be restored on the next update cycle.
      • 12:00. The MPCORB files have been restored. The problem also affected, to a much lesser extent, the MPChecker files, and these are being rebuilt.

    • Webserver issues II
      2008 Feb. 26: 10:00. A number of observers have reported erroneous junk rating values in their acknowledgement messages. This problem is being investigated.
      • Feb. 27: 10:00. This problem is believed to be fixed.

    • Webserver issues
      2008 Feb. 25: 11:10. A number of users have reported (intermittent) problems with "file protection" messages appearing when using web services. There appear to have been some issues with protections on directories when the webserver was moved over to the new machine. The reported permanent problems appear to have been resolved.

    • Webserver move (Feb. 19)
      2008 Feb. 14: 15:35. The MPC webserver will be moving to a new, faster computer. The current webserver will need to be shutdown at noon, to allow a clean copy of numerous files to the new machine. AUTOACK will also be disabled at the same time. It may take several hours to transfer all the necessary files. Once the new webserver is restarted users should see no difference in accessing web services. There may be delays in propagating the necessary DNS change--this is out of our control.
      • Feb. 19: 10:40. The automated processing queues have been stopped.
      • 11:47. The AUTOACK process has been stopped.
      • 12:00. The webserver has been stopped.
      • 14:10. All necessary files appear to have been moved to their new locations. Waiting for local DNS to update before restarting webserver.
      • 15:00. AUTOACK process has been restarted.
      • 15:30. Automated processing queues have been restarted.
      • 18:00. Webserver still off-line as required DNS changes not yet made.
      • Feb. 20: 10:48. Have restarted the old webserver as no sign of DNS changes being made.
      • 13:30. DNS changes have been made, visible internally. Apparently change was visible externally around 12:00.

    • CF mail server outage (Feb. 21)
      2008 Feb. 14: 15:31. We have been informed that the CF's SMTP server will be rebooted at 07:00 EDT on Thursday, Feb 21. During the expected 30-minute outage, incoming mail (e.g., to mpc@cfa) will not be received, but the mail should be resent by the remote systems when the SMTP server is restarted.

    • Galaxy names in SNChecker
      2008 Feb. 6: 08:39. A user reported that the SNChecker was apparently not resolving galaxy names correctly. A check showed this to be the case and a modified executable was put on-line.

    • Missing Objects in MPChecker
      2008 Feb. 4: 14:39. A user reported that an object was missing from the output of MPChecker. An investigation showed that the file of orbits used by the MPChecker service had not been built correctly by the overnight procedure due to two processes trying to write to the same file. This problem was believed fixed some months back. It is now fixed.

    • DOU MPEC
      2008 Jan. 19: 14:00. Last night's DOU MPEC failed due to a malformed orbit in one of the files crashing one of the check programs. The check program has been recoded to prevent a similar occurence in the future from crashing the program. The offending orbit has been removed and the bits of the DOU MPEC that failed are being rerun.

    • Special-Epoch NEA*.DAT Files
      2008 Jan. 12: 22:00. A user reported that the elements for 2008 AF3 in the special-epoch NEA*.DAT files seemed incorrect. Investigation showed that the same unchanged elements are used in all the NEA*.DAT files. The same erroneous behavior occurs for a number of other objects. The reasons for this problem are as yet unclear. It is possible that it is related to the Dec. 12 problem listed below. Further investigation will be undertaken after the update tonight associated with the DOU MPEC. Until this problem is resolved, users requiring NEA elements for non-standard epochs are advised to use the MPES to obtain this information.
      • Jan. 13: 17:35. A problem with a bit of rarely-executed code in the program that generates the NEA*DAT files has been located and, we believe, fixed. The update associated with tonight's DOU MPEC should remove the problem with some of the elements in the NEA*.DAT files. This problem affected only the NEA*.DAT files, not any primary orbit files.
      • Jan. 14: 06:20. A new set of NEA*.DAT files were put on-line around 02:30. The new set is based on orbits from the previous DOU MPEC: a set based on orbits from the current DOU MPEC is currently being run.
      • Jan. 14: 08:40. The NEA*.DAT files are now based on orbits from the latest DOU MPEC.