• Processing (Info)

    • MPC Status Page: Archive (2004 July-December)

    This page describes enhancements to or problems that have occurred with the MPC's webpages and scripts and the fixes that have been made.

    Recent problems are listed elsewhere. Index of other older problems..

    Older Enhancements and Resolved Problems

    • Telnet-based Extended Computer Service
      2004 Dec. 26: 13:30. The telnet interface to the MPCUPDATE service has been retired.

    • MPES
      2004 Dec. 23: 15:38. A number of enhancements have been added to the MPES. Both the features listed below apply only to objects with uncertainty maps, these are typically one-opposition NEOs.
      • Access is now possible to the variant orbital elements used to generate the plots and lists of uncertainty regions. To request the n-th variant orbit for a particular object put the required number after the designation enclosed in square brackets: e.g., "2004 EC104 [23]" will return variant orbits number 23 for 2004 EC104. Residual blocks are not available for these variant orbits and no uncertainty information is displayed.
      • The size of the major axis (in arcseconds) and the position angle of the three-sigma uncertainty are now displayed on each ephemeris date. This feature is intended for current dates only and the information is suppressed for dates more than 400 days away from the current standard epoch.
      This information will work its way into the PDF documentation in the New Year.

    • Web-based Extended Computer Service
      2004 Nov. 25: 13:30. The Web-based ECS has been extended in a number of ways:
      • The MPCAT-OBS service now includes data files containing observations of Pluto and the outer irregular natural satellites of the giant planets.
      • The MPCUPDATE service now includes a link to download a zip file containing all the data files associated with a particular batch of MPCs.
      These changes are part of an on-going effort to retire the telnet-based computer services.

    • MPES
      2004 Nov. 24: 12:10. There has been a long-standing minor difference in the MPES regarding the dates of last observation and desirability of further observations between the full output and the summary output. This difference has now been removed.

    • CBATMPC Upgrade
      2004 Oct. 8: 14:20. The MPC/CBAT has one Unix box, cbatmpc, which is administered by the CF and which we use primarily to e-mail out the MPECs and IAUCs. The CF started a planned OS upgrade on this machine at 07:00. The first upgrade didn't take, so they tried again. That didn't take. There are apparently problems in completing the upgrade. The machine is currently in an unusable state. If it cannot be fixed today, tonight's DOU MPEC will have to be abandoned. The next mid-month MPS batch may also be affected if the machine is not back online by tomorrow night.
      • 14:47. We are now informed that we should have the machine back on-line today.
      • 15:08. We have just heard that cbatmpc is back on-line. Assuming no problems with the upgrade, DOU MPECs and mid-month MPS batches should be issued as normal.

    • MPCORB Mirror
      2004 Oct. 8: 11:00. A user remarked, not to the MPC but on a mailing list, that the MPCORB mirror site in the Czech Republic had not been updated since Sept. 8. An attempt to run the update script manually resulted in an odd message. Consultation with CF followed. It turned out that when one of the CF boxes was upgraded on Sept. 8, the version of SSH2 was also upgraded. Unfortunately for anyone trying to do any real work on the system, the command-line flags for the new version of SSH2 were incompatible with the command-line flags on the original version! Yet the authors of the new incompatible version retained the same command name! Fortunately, the fix is simple, and the MPCORB files have been updated.

    • Full Disk and AUTOACK
      2004 Oct. 8: 10:00. A user disk filled up overnight. Observers will have received bouncebacks from their submissions, but, as noted in the Guide to Minor Body Astrometry, the observations will still have been received by the automated processing routines.

      An unrelated problem caused AUTOACK to stop processing at around the same time the disk filled up. A fix has been made to a library routine to try and prevent this particular problem happening again.

    • Sending of Designations
      2004 Oct. 1: 13:10. We are still having problems sending out designation batches to observers in a consistent manner. This is a result of the CF's policy on blocking direct SMTP access to observatory machines. Some of our machines are an exclusion list to the above policy. We want the exclusion list extended to all our machines, the CF wants it emptied. One staff member's workstation is not on the exclusion list and consequently many designation batches have not made it out from his machine. In the coming days, we will be moving over from a per-user scheme for handling designations to a system-wide scheme. This should alleviate the problem.

    • Missing files in the Daily MPC Update in the ECS
      2004 Sept. 28: 15:00. A user reported that the pages listing the files associated with each DOU MPEC had not been updated since Sept. 10. This is another result of the new security policy. The procedure that copies these files over to the webserver has been fixed, but the files for days 255 through 271 (although listed on the index.html page) are not available.

    • AUTOACK problemette
      2004 Sept. 25: 20:05. As a result of a rewrite of the program that formats incoming observation batches to take advantage of new subroutine to parse telescope details, the AUTOACK program (which accepts the output of the formatting program as input) now has no access to the original TEL line submitted by the observer. This could lead to misleading information on formatting compliance being returned to the observer. A fix is being worked on.
      • 20:22. The AUTOACK program has regained access to original TEL line. Telescope formatting compliance messages should now be as expected.

    • Disk full problems
      2004 Sept. 24: 09:50. A data disk apparently filled up overnight, causing some problems with filing operations on orbits for the MPES. The affected disk has been cleaned up, the failed routines are being rerun and the missing orbits should reappear shortly.
      • 10:15. Normalcy is apparently restored. Please report any problems asap.

    • More "Security" Edicts from the Computation Facility II
      2004 Sept. 8: 17:30. As feared, a number of routines that copy material over to the webserver failed to get changed. On the whole, the change in routine has gone without too many problems. If you notice problems, please report them asap.

    • Non-mailing of recent MPECs
      2004 Sept. 3: 12:00. It seems that the CF box that mails out MPECs decided to stop sending them yesterday. Attempts to get the automated mailing working again are not yet successful, so the back-logged MPECs have been sent out manually.

    • Lists of NEOs
      2004 Sept. 1: 12:00. The lists of unusual objects have had the names of the discoverers suppressed. There is an apparent problem that recently-discovered objects were being credited to the wrong observers. It is unclear whether this problem is related to the fact that the MPC is currently in "preparation" mode. The discoverers names will reappear when the problem is fixed.
      • Sept. 3: 09:00. The discoverer details are now correct, suggesting the problem is related to being in "preparation" mode. We shall investigate.
    • MPEC 2004-Q57
      2004 August 28: 12:00. It seems there was a production problem with last night's DOU MPEC that prevented the new NEO observations from being included. We are investigating, the observations should appear on tonight's circular.

    • More "Security" Edicts from the Computation Facility
      2004 August 26: 15:40. We have received notification from the Computation Facility that all passwords to their CF boxes must be changed before Sept. 7 and that they must now be changed every 90 days. The MPC has numerous command files that copy material over to the CF webserver. Most of these are executed by entirely automatic processes that run at intervals varying from minutes to months. Some command files are generated dynamically, others are static. The dynamically-generated command files are not affected by this edict. The static files will have to be modified to be dynamically generated each time the relevant processes are run. This is a major headache and the changeover will probably take several days to accomplish. It is possible that some command files will not get changed (or will get changed incorrectly--it isn't always possible to test run a modified command file), so some pages may not update properly after September 7.

      This author's opinion is that this change will actually reduce the level of security on the CF boxes, as users may now have to begin writing their passwords down instead of committing them only to memory. That is what I'm going to have to do...

    • Minor Planet Checker
      2004 August 19: 12:00. It seems that a nightly procedure to update the element files used by the Minor Planet Checker has been silently failing possibly for as long as a month. This means that newly-generated orbits are not being included in the search. The missing elements are being reinserted into the element files.

    • New observer options in MPES and EOSJT
      2004 August 15: 23:30. A number of new observer options have been added to the MPES and the EOSJT: suppression of ephemeris data when sun is above local horizon; suppression of ephemeris data when object is below local horizon; and allowance for quick selection of non-0h-UT daily ephemerides.

    • Patch installation on cluster
      2004 August 13: 10:30. Today we are installing a number of patches on each of the cluster machines, in order to clear up a number of minor problems (none security related...) with the latest version of the OS. There will be brief outages (hopefully under 30 mins a time) of the webserver (and other machines on which it relies) throughout the day.
      • 12:47. The last of the machines was patched at 12:38, ahead of schedule.

    • Perturbed NEA elements for non-standard epochs
      2004 August 7: 12:00. A user reported that a known NEA was 30' off the prediction returned by MPChecker. An investigation showed that the routine that calculates elements for non-standard epochs was occasionally outputting incorrect values for certain types of orbit when requesting element output at widely-separated intervals. This has now been fixed. Since MPChecker uses elements at the current 40-day date generated by this program, this problem manifested itself by degrading MPChecker's output. The standard-epoch orbits from last night's DOU MPEC are being rerun through the corrected program. The files used by MPChecker will get rebuilt as part of this procedure.
      • August 8: 12:00. NEA orbits published on MPECs since the last full MPC batch are being rerun through the corrected code to fix the problem. Just to make it clear, the orbits as published on the circulars are correct, the problem lay in elements prepared for certain non-standard epochs.

    • Local circumstances for geocentric orbits in NEOCP
      2004 July 21: 20:00. The calculation of local circumstances for geocentric orbits in the NEOCP is not correct (I knew there was a reason that block of code was commented out...). Until the program is fixed, please ignore the local circumstances. I've only just got back home and I have to eat before I attempt to modify the code.
      • 21:00. Having eaten, I have now fixed (I think) the code.

    • Network outage July 21
      2004 July 21: 12:35. The main UPS in the CF died on 14 July and since then the CF machines and network routers have been running on Cambridge power (residents will understand why this is not desireable, particularly in the summer months...). They are planning to shut down the CF at 17:30 EDT today, in order to put everything back on the UPS. They expect to be back up by 19:00. All MPC services will be unavailable during this window.
      • 19:45. Normal service has been resumed.

    • Sluggish response from CfA webserver
      2004 July 19: 20:40. We've noticed that the CfA webserver is very sluggish, as is our webserver (we see a long pause between entering form data and clicking the action buttion and the cgi script beginning running). We are investigating with the CF as to whether this is a general problem with the network. We do not see a problem invoking processes on our webserver machine (which suggests that the problem is not something local to our system). We have just shutdown then restarted the webserver, but that didn't help.
      • 20:50. The copying of this HTML file to the CfA webserver was very slow, further indicating some network problem or computer problem in the CF.
      • 21:55. No response from the CF. Network activity within our cluster appears to be normal. However, there are significant delays in running commands on the CF machines (e.g., 40 seconds before getting the menu screen after typing 'pine').
      • July 20: 00:30. Still no response and the problems persist.
      • 08:40. E-mail to @cfa.harvard.edu address is being to trickle in. Perhaps the problems are being fixed. We'll find out when we get into the office.
      • 09:56. Arrive in office to find e-mail being delivered and better response from both webservers. Wonder whether we'll get an explanation for the problems?
      • 11:00. It seems that the earlier network problems weren't quite as fixed as was hoped. Replacement of a CF network switch is pending. Hopefully, this replacement will not affect our cluster.
      • 13:47. The CF has informed us that the faulty network switch has been replaced.

    • Planned network disruption July 17 III
      2004 July 17: 13:00. Well, we didn't get an OK from the CF, but I went down there at 12:20 to enquire about the status of the investigation only to find no-one there! In the hope that if the problems weren't fixed the CF staff would not have gone to lunch/home, we are assuming that the problems are fixed. One of the shutdown machines was brought on-line as a test. This successfully rejoined the cluster, so all the remaining machines were brought back on-line. Normalcy should have been restored.

    • Planned network disruption July 17 II
      2004 July 17: 08:00. We have begun to prepare for shutting down most of our cluster machines prior to the pre-announced network outage.
      • 08:50. Shutdown of all but four machines achieved from home.
      • 09:30. Heading into office to await all clear to restart machines.

    • No mid-month MPS batch this week II
      2004 July 16: 22:10. Assuming that the CF investigation tomorrow is satisfactory, we have decided to proceed with this week's mid-month MPS batch, albeit with a day's delay.

    • Planned network disruption July 17
      2004 July 16: 16:54. The CF has announced that they are planning downtime of the CfA network from 09:00 to 12:00 EDT tomorrow (Saturday) morning in order to check that today's fix has actually solved the network problem (flooding of network traffic due to a router loop). We intend to power down our machines shortly before 09:00 to ensure that they are brought down in a controlled fashion. They will be brought back on-line as soon as possible after we get the OK from the CF.

    • Network disruption July 16 part II
      2004 July 16: 16:30. Another network outage has occurred. The CF apparently got good logs from the outage this morning and are trouble-shooting with Cisco. A faulty router has been replaced. Our machines are back up, but we need to restart various automated procedures.
      • 16:50. The restarts have been performed. Normalcy should have been restored.

    • No mid-month MPS batch this week
      2004 July 16: 15:30. In light of continuing network stability problems, the decision has been made to not issue a mid-month MPS batch this week. The preparation of this journal is extremely network intensive and the failure of the network at an inopportune moment in the preparation would require many hours of fixing. In addition, the preparation occurs over a weekend, a period when CF response to problems is slower than during the week. Network willing, we hope to resume normal service for the next issue.

    • Network disruption July 16
      2004 July 16: 09:30. Another network outage has knobbled our machines. No information from the CF as to the reason for the outage, or, for that matter, any notification that an outage occurred. Machines will become available again as soon as we have some confidence that the network will not die immediately.
      • 11:00. We have restored normalcy. Again. No guarantees from the CF that the problem will not reoccur.
      • 11:50. It turns out that the DOU MPEC did not complete running. It seems there were at least two network outages last night, the first around 03:00 and the second sometime after 08:00. The bits of the DOU MPEC that did not run are now being completed.
      • 12:00. A number of other overnight jobs did not complete and these are also being rerun.

    • MPCs in ADS
      2004 July 15: 20:00. We have noticed that recent issues of the MPCs are not indexed in the ADS. This has apparently been caused by the ADS mail system rejecting our e-mails containing author and title information for recent batches. We are investigating this problem with the ADS staff.

    • Telnet computer services on CFAPS8
      2004 July 14: 15:00. The recent loss of a data disk on the old VAX that powers one half of the telnet computer service has caused us to retire this functionality, at least temporarily, on CFAPS8. Some of the facilities offered on CFAPS8 are also accessible on CFAPS1. Most of the telnet-accessible features have, in any case, been replaced by features on the web service.

    • Network disruption July 14
      2004 July 13: 16:58. We have been informed that the CF will be replacing a faulty network device on July 14 from 06:00 to 08:00 EDT. We have been warned to expect network instability during this period. Machines (such as the webserver) may not be accessible for extended periods during this time frame.
      • July 14: 09:40. The network caused numerous connectivity problems in the cluster, as expected. As we clearing a number of remaining problems the network froze again--there was an extended power outage in the CF.
      • 11:30. After trekking into the office, it seems that normalcy is restored.

    • Minor Planet Checker
      2004 July 13: 21:30. A user reported that the Minor Planet Checker was reporting inconsistent offsets for an NEA when pasting in observations. Investigation showed that the program was doing a geocentric computation, ignoring the observatory code on the entered observation. This was caused by a failure to modify the arguments on a library routines following an upgrade. This has now been done.

    • Network loss?
      2004 July 11: 11:00. Early this morning, the machines in our cluster lost their connectivity. The first machine went off-line and rebooted around 00:30. Over the next eight hours, all but one of the other cluster machines rebooted. This is presumably network related and similar to the problems of July 6 (we suspect this as a home connection to one of the CF's boxes was also disconnected). Unfortunately, this interruption has caused havoc with our nightly publications (DOU MPEC and the mid-month MPS batch). We are working to restore the missing publications.
      • 11:30. It has been determined that the DOU MPEC preparation routine did not even begin running. This morning's DOU MPEC is therefore abandoned.
      • 11:42. The SMTP queues on six machines that did not restart automatically have been restarted.
      • 11:45. The bits of the mid-month MPS preparation routine that did not run are being run.

    • Network loss
      2004 July 6: 14:00. Late this morning, the network at the Observatory died. Most of our cluster members lost their connectivity. No explanation for the outage is as yet forthcoming. We have restored normal service.
      • 15:00. Our desktops again froze around 14:30. Normal service was restored after about 20 minutes.

    • Blank e-mail and web version of the DOU MPEC
      2004 July 4: 09:15. A blank version of MPEC 2004-N20 was e-mailed out this morning. The correct version has been sent out.