MPC Status Page: Archive (2012 January-December)

Processing (Info)

MPC Status Page: Archive (2012 January-December)

This page describes enhancements to or problems that have occurred with the MPC and scripts and the fixes that have been made.

Recent problems are listed elsewhere. Index of other older problems..

Older Enhancements and Resolved Problems

December 28 MPCAT-OBS Comet File Replaced
2012 Dec. 30: 21:50. Following a report from a user that the MPCAT-OBS file of comet observations was the same as that from the previous month, the file has been replaced.
December 28 DOU MPEC
2012 Dec. 28: 06:00. Tonight's DOU MPEC will not be issued, to avoid any possible conflict between its processing and the processing associated with the filing of the Dec. 28 MPC batch which is still running.
Malformed MPCAT-OBS Files (December 19)
2012 Dec. 19: 23:45. A number of users have informed us that the mid-month complete observation files associated with the 2012 Dec. 16 batch of MPS are corrupted. We have tracked down the reason for this corruption and are regnerating the files.
- Dec. 20: 03:53. Corrected files have been put on-line.
SMTP E-mail Interruption (December 18)
2012 Dec. 17: 16:45. We have been informed that the CF's SMTP server will be off-line for approximately 60 minutes beginning at 20:00 UTC on Tuesday, December 18, to allow replacement of a faulty CPU. Attempts to send e-mail to us during this period will fail, but the mail should queue up on the sending computer for a subsequent resend attempt.
Missing Objects in MPChecker
2012 Nov. 1: 16:00. The previous problem of missing objects in the MPChecker service has resurfaced. We are investigating.
Fixes for a Number of Problems
2012 Oct. 23: 15:00. A number of fixes for minor web-based problems have been made.
- The lists of Trojan objects are working again.
- Recently-discovered comets are accessible in both the MPES and MPChecker.
- The complete file of skycov pointing data has been moved from the CF anon-ftp site to our own web server. The updating of this file is now on a monthly schedule.
Last Night's DOU MPEC
2012 Oct. 17: 14:00. Last night's DOU MPEC has been abandoned due to an internal problem.
Power and Network Problems
2012 Oct. 14: 10:00. There are problems with the MPC machines over at 60 Garden Street. We will investigate as soon as possible.
- Oct. 15: 03:00. A UPS has failed. This caused one machine to go off-line and three others to reboot. In addition, the main switch controlling the small intra-net is having problems: it seems to be resetting itself every 10 seconds, which is preventing the rebooting machines from coming up cleanly.
- 04:00. The shadow sets are rebuilding, we'll see what progress has been made in the morning.
- 09:30. No progress on the shadow sets. The cluster is being rebooted. Ensuring that nothing is left from incomplete runs of automated processes will take a while.
- 15:00. The final bits are being restarted.
MPEC 2012-O37 Resend
2012 July 31: 19:37. MPEC 2012-O37 has just been remailed. The initial mailing failed: the VMS side handed the necessary files over to mpcapp1; mpcapp1 handed off the relevant files to the mail system; the mail system handed off the files to the Harvard mail server; then nothing. We will try to find out what went wrong on the Harvard side.
Temporary Move of Webserver
2012 June 26: 19:56. The MPC webserver mpcweb1 is having issues. While we investigate (and fix!) these problems, the webserver has been moved to our database machine, mpcdb1 (131.142.195.57). A temporary change has been made in the DNS records, but this change may take time to propagate. Users should continue to access URLs at www.minorplanetcenter.net as usual.
- 20:18. We now see the new IP addresses on our home systems.
- June 27: 03:07. We are moving the hostname www.minorplanetcenter.net back to mpcweb1.
Customize Page
2012 June 15: 03:41. A user reported a problem with the Customize page script. Incomplete pages would be returned. The problem has been identified and fixed.
Time Issue on Scully
2012 June 6: 16:18. The system time on scully seems to be misconfigured. Displaying the date and time shows a UTC time, but the timezone is indicated as EDT. This is confusing the cgi scripts.
Problem With (Some?) Odd-Epoch Elements in MPES
2012 May 18: 15:30. As a result of a MPC member noticing a message detailing a problem with an MPES ephemeris on a mailing list, it has been determined that some comet elements may have been incorrectly integrated to a current epoch. The use of a current epoch in the MPES is intended to allow observers to generate their own unperturbed ephemerides, with reasonable results. The problem seems to be related to the recent procedure (running under Linux) introduced for producing odd-epoch elements. As a test, we are regenerating the odd-epoch comet elements using the old method (running under OpenVMS).
- 17:47. The offending data file has been replaced. The correct elements and ephemeris are now returned in the MPES. The orbital elements for software packages have been regenerated. We are trying to reproduce the error on the Linux side that caused this problem, but have not yet succeeded.
Webserver Rebooted
2012 Apr. 30: 18:15. The MPC webserver machine has been rebooted, due to another critical memory shortage, presumably caused by a memory leak. The condition was flagged by a monitoring tool, but we are investigating the causes for the leak.
- May 3: 14:30. We have identified a potential cause for the reboots. It seems that certain search engines are ignoring the robots.txt file and are pounding the DB with thousands of requests that return large amounts of data. The IP addresses (and sub-domains) of these problem search engines have been blocked. Further blocks may be put in place as needs arise.
Webserver Rebooted
2012 Apr. 23: 14:37. The MPC webserver machine has been rebooted, due to a critical memory shortage, presumably caused a memory leak. The condition was flagged by a monitoring tool, but we are investigating the causes for the leak.
Network Problems at CDP?
2012 Apr. 18: 02:40. We are receiving reports that MPC webpages and services are unavailable. The public-facing machines are up and running. We are not yet sure what the problem is, but are investigating.
- 04:10. The webserver machine has been rebooted. Pages are being served as normal.
Sending of Designations II
2012 Apr. 10: 01:30. The datafiles necessary to allow the resumption of the AUTODES service have been recovered from the failed system. The automatic sending of designations has resumed.
Recent MPEC non-mailing
2012 Apr. 1: 15:00. Recent MPECs do not seem to have been mailed out via mpcapp1. The affected circulars are numbers 2012-F107 to 2012-F114, and 2012-G01 to 2012-G06. The necessary files for each mailing have been copied from the VMS side to the Linux side (confirmed by examination of the VMS log files) and the circulars are up on the website, so the VMS side is working normally. The Linux system thinks that they've been e-mailed, since the files associated with each mailing that get deleted after the mailing have been deleted. We are investigating. The DOU MPECs are unaffected by this problem, as they continue to be mailed out via the old route. Use of the new route for the DOU MPECs, which was planned for the next few days, will be delayed.
- 16:15. MPECs 2012-G08 and -G09 have been e-mailed without problem. No modifications were made to the mailing process on the Linux systems. The missing MPECs will be re-mailed later today.
- Apr. 2: 03:06. All the missing MPECs have been remailed.
- Apr. 11: 15:05. The DOU MPEC is now being issued via the new route.
Multiple e-mail copies of MPEC 2012-F75
2012 Mar. 23: 19:05. The first MPEC (issue 2012-F75) has just been issued via the new mpec@mpcapp1 route. The script that issued this circular was run several times, since it appeared that nothing was happening and that the script had crashed. This was apparently not the case and multiple copies of this circular were sent out. There should be only one e-mailed copy of MPEC 2012-F76.
Sending of Designations
2012 Mar. 16: 16:45. The failure of a machine has caused the automated designation sender to cease functioning. It may take some time for this service to resume operation, as we have to recover data from a disk.
Network Outage Planned (Thursday Feb. 9)
2012 Feb. 8: 15:40. We have just been informed that there will be a two-hour network outage at CDP (where the MPC is now based) from 09:00 UTC (04:00 EST) on Thursday, February 9. This outage is to allow an upgrade of the network router that connects CDP to the CfA site. There will be no network connectivity to/from CDP during this window.
Missing Orbits in MPES
2012 Feb. 8: 14:30. We have received a couple of reports of problems accessing orbits in the MPES. The problem has been traced to an incorrect build of a monthly data file. The affected file is being rebuilt.
- 19:00. The problem has been fixed.
Problem accessing www.minorplanetcenter.net
2012 Jan. 28: 01:30. A number of people have reported getting no response from www.minorplanetcenter.net since about 00:20. We see the same behavior. The machine is up and accessible. It appears to be a routing problem within the Harvard network. We are investigating.
- 02:30. The problem seems to have resolved itself.
January 26 DOU MPEC
2012 Jan. 26: 13:30. The January 26 DOU MPEC was abandoned due to a failure early on in the production procedure.
January 12 DOU MPEC
2012 Jan. 12: 16:00. The January 12 DOU MPEC was abandoned due to a failure early on in the production procedure.
CfA Network Problems?
2012 Jan. 11: 03:20. We are experiencing network problems in accessing our work accounts from home. We are receiving reports that outside users are experiencing issues accessing the MPC web site. We are checking whether there are network problems at CfA with our Computation Facility. Access was restored after some hours. No explanation for the problems has been forthcoming.