January 13, 2012

Our mail server, hermes will be down for a few hours tonight (1/13/2012) for upgrades that we believe will improve service. No received mail will be lost, but queued during that period and delivered when the upgrades are complete.

December 28, 2011

The UGCS sysadmins would love constructive feedback about our services. Feel free to email us as sysadmins at ugcs dot caltech dot edu.

December 12, 2010

Earlier this morning, mail delivery was held up for several hours. The mail queue has now cleared, and all messages sent should be delivered.

November 16, 2010

We will have a maintenance window on Saturday, November 20 from 9am to 4pm. During this time, various UGCS services may be down, and we will be restarting all login servers.

October 15, 2010

We have finished the rollout of Debian Squeeze on all of our shellservers and mortals. We think we have a complete development environment for most languages on them- but if you find something not there that you want, let us know and we'll get it installed.

September 13, 2010

We are in the proccess of moving our shellservers and mortals to a more recent version of Debian ("squeeze"), so please let us know if there are any bugs that need to be fixed. Login may be unstable for the next day or two as we reboot the machines to load the new images, but they should be stable after that.

August 13, 2010

One of our AFS servers (apollo) failed to reboot around 11am, causing an AFS outage. Mail bounced for about 20 minutes, but none was silently dropped. As of 12:30pm, things are on their way back up.

August 8, 2010

Air conditioning failure in the UGCS server closet caused a number of difficulties. Apollo crashed, taking the AFS root volume with it, and so homedirs, the website, and mail were temporarily unavailable. Furthermore, the mortals subcluster was turned off to save heat. All mail should have been now delivered, and homedirs should now be available. For further information or questions, email the sysadmins.

June 21, 2010

An unexpected power outage in Winnett from 2:00 to 6:00 AM resulted in massive service disruption (aka the whole cluster was down) for about four hours. Any mail sent during that time was collected offsite and has since been delivered. All services and servers should now be up and running. If you have any questions, please email the sysadmins.

June 17, 2010

We've moved all the shellservers to a new 64-bit image, and the lab has been cleaned and reorganized. There are some small changes to the new image that need to be made, so if you notice anything out of place please email the sysadmins.

May 23, 2010

We now have two machines, terpsichore and minthe, that have high-performance graphics cards (Nvidia GTX 260), as well as 4gb ram and a quad-core processor. These computers were purchased with a grant from the Housner Fund

April 13, 2010

Office hours this term will be be Mondays from 10-11:30pm.

March 22, 2010

We will have some downtime on Monday, March 28 starting at 12:01am and lasting until 6am at the latest.

January 6, 2010

Office hours for this term will start next week and run from 7:30-9pm.

December 2, 2009

Office hours tonight will run from 7:15-8:00pm.

November 4, 2009

The Mortals cluster is back up.

October 18, 2009

We had some brief downtime on posiedon, our webserver, this morning from 9:30 to 11:30. All services should be back up- please let us know if you have problems.

October 6, 2009

We are going to start holding Office hours in UGCS. Current office hours times are Wednesdays 7:30pm - 9pm, but are subject to change- see the page for updates. Come by to say hi, get your password reset, or ask questions about Linux or UGCS.

September 27, 2009

We are currently migrating to Postgres 8.3. For portions of tonight, postgres may be down, and changes made may be lost.

Update: 11:59pm The migration should be complete, and databases should connect to 8.3 by default.

September 13, 2009

We will be migrating to Postgresql 8.3 in a few weeks. To help you make this migration, we have set up a Postgres 8.3 server at postgres:5433. Please test your applications against this new database. See Postgres migration for more information. As always, contact us if you have any questions.

August 22, 2009

A "phising" email was just sent to many UGCS users which asks you to reply with a username and password. Do not reply to this message- it is a scam. UGCS sysadmins will never ask you for your password.

August 19, 2009

We have set /afs/.ugcs/ugcs-admin to be world-readable. You can now look through the various utilites we have written to help automate the cluster administration. Over the next few months, we will be setting up a website to better document the code we have written.

We also have a read-only copy of our svn repository available at, or you can access it at file:///afs/.ugcs/ugcs-admin/svn We're quickly adding our projects to it, so check back often!

August 14, 2009

Over the next day or two, we will be rebooting our shellservers with a new setup. This will improve their speed and usability, but will result in some downtime in various shellservers. Contact us if you're planning on running and long-term jobs. <math>Insert formula here</math>

June 21, 2009

The outer door has been fixed by "facilities".

June 20, 2009

The outer door to UGCS is jammed closed, making the lab inaccessible. If you are trapped inside, please send us an email or call security at 626-395-5000 (extension 5000 on campus). Otherwise, the lab will not be available for a few days. We will work with Facilities to take care of this as soon as we can.

May 5, 2009

UGCS is running its annual sysadmin search. If you are interested in being a UGCS syadmin, please take our sysadmin survey.

April 3, 2009

All user principles have been modified to allow Kerberos tickets to be requested for up to 365 days. Tickets may be renewed up to a total length of 3652 days. Unfortunately, Kerberos does not allow for an unexpiring ticket, analogous to an SSH keypair, but these should be close enough.

Febuary 18, 2009

UGCS now supports svn protocol for subversion source control hosting. We may add additional source control programs if users express interest (eg, git). For more information, see the page on Subversion.

Additionally, we are now running a PGP keyserver, for those of you who make use of such encryption. You can find it at [[1]], [[2]], and [[3]].

October 29, 2008

We will have a UGCS sysadmin meeting on Sunday, November 2, at 5pm in the UGCS lab. All are welcome to attend. See the UGCS Blog for more information.

July 28, 2008

We had an unplanned mail outage last night, and mail was not being delivered. We have fixed the problem and the mail queue should finish flushing with a few hours.

May 22, 2008

There will be scheduled downtime due to Caltech needing to bring down power for a few hours on Friday, May 23 from 4:00 AM - 6:00 AM. We will restore all systems as soon as possible after this event. Our UPS is kind of wimpy, sorry.

May 21, 2008

UGCS is back up after a several-hour power outage. All services should be back to normal- please let us know if you have any problems.

May 13, 2008

Due to a recently discovered vulnerability in OpenSSL, all UGCS host keys will be changing shortly. We will post a GPG-signed list of the new keys when we are done. It is a good idea to download this list and add it to your ~/.ssh/known_hosts file to prevent man-in-the-middle attacks.

Update: Host keys for our login machines have been changed. A GPG-signed list of the key fingerprints is available at

May 13, 2008

Several UGCS, alumni, and IMSS users have been reporting increased phly phishing attempts (IE, trying to get you to send them your username and password and claiming to be the admins). Just a reminder that the UGCS sysadmins will never ask you for a password. If we want to confirm your identity we may send you a long string of letters and numbers and then ask you to send it back.

If you're more paranoid, all official email from sysadmins will be signed by that sender's GPG public key. If we forget, ask for a resent signed version before taking any action. You can find the UGCS Sysadmin key as well as our individual keys at the CA page.

May 11, 2008

The Mortals sub-cluster has been added. Enjoy!

March 18, 2008

We have blocked all Kerberos traffic that does not come from UGCS to help mitigate a recent Kerberos vulnerability.

Update We have patched the vulnerability and non-UGCS Kerberos has been unblocked. 13:14, 18 March 2008 (PDT)

March 6, 2008

Due to unexplained issues with hera (our main LDAP server), we're routing all LDAP requests through to zeus (the Kerberos head and LDAP backup). As a result, things may be slower and less reliable. We are investigating the issue and hope to have it fixed soon.

March 2, 2008

Like Linux? Want to help make UGCS a better place? Think you've got what it takes to be a UGCS sysadmin? Well, then you're in luck. UGCS is conducting its annual sysadmin search. If you're interested, please take our Sysadmin Survey and submit it by the end of spring break (March 30).

February 21, 2008

In order to complete an upgrade to our mail server, mail services will be down for about an hour beginning around midnight.

Update Mail delivery is back up 01:22, 22 February 2008 (PST)

February 13, 2008

We have automated the mailing list creation process. To create a mailing list, login to UGCS, and run the command create_mailinglist <LISTNAME>. Your list will be created within the minute. If you should run into any problems email me. 18:15, 13 February 2008 (PST)

February 10, 2008

CGI scripting and PHP has been disabled due to a serious new kernel vulnerability. We are working on fixing this so that that we can re-enable scripting on our webserver.

Update: Scripting on poseidon has been re-enabled. 15:17, 10 February 2008 (PST)

February 5, 2008

We are currently experiencing unknown problems with mail delivery. You can still access your mail on UGCS, but our server is currently not accepting new mail.

Update: Mail delivery is working again 03:38, 5 February 2008 (PST)

January 4, 2008

Happy New Year! Unfortunately, we experienced difficulties with our virus scanning and spam blocking systems that resulted in mails being queued for delivery between January 2 and January 4 being held. We are currently clearing the backlog and all mails should arrive at their destinations shortly; no mail was lost.

Update: The mail queue finished flushing by 9pm. All mail has been delivered and mail delivery is back to normal.

November 4, 2007

The maintenance scheduled for the evening of November 3 has been completed successfully, resulting in greatly increased performance on AFS transfers, mail delivery, and LDAP queries. Additionally, we have deployed a new application security framework in order to proactively detect and block SQL injections and compromised guestbook/bulletin board applications. mod_security may have side effects for your CGI applications, so please don't hesitate to contact us if your scripts stop working.

November 1, 2007

We are planning some downtime for the afternoon and evening of Saturday November 3 in order to install gigabit ethernet cards in our mass fileservers and bring additional backup servers online. There may be disruptions to your ability to access files in your home directory, although we will try to minimize impact as much as possible.

Update: the downtime should be largely finished, except for rolling reboots on our shellservers. Apologies for the disruption to file and e-mail services, a few things didn't go as planned, but everything is back up now, and performance should be vastly enhanced. 21:32, 3 November 2007 (PDT)

October 25, 2007

2:20am: We appear to be under an smtp DDoS attack. We are taking measures to mitigate this attack. To reduce ldap load, our ldap database has been set readonly, so any changes to user information will not work. Also as a preventative measure, we have added more restrictive filtering rules for our core servers- including blocking non-Caltech Kerberos requests (this will be unblocked in the morning). If you notice that we have blocked a critical service, please contact us. We will post updates here as we have them 02:25, 25 October 2007 (PDT)

Update All services should be back to normal 11:13, 25 October 2007 (PDT)

October 24, 2007

The new UGCS job posting system is up. New jobs postings should be added to /afs/ugcs/drop/jobs. If you want a listing removed or changed, please contact a sysadmin. More documentation on the job system can be found here.

October 24, 2007

We recently needed to restart a large number of services due to the openssl 0.9.8g update. If you notice any problems, please contact us. Additionally, we experienced 3 hours of website homedir downtime on the afternoon of the 23rd due to an AFS readonly mirror bug. We have disabled the readonly mirror behavior until we can investigate and things appear to be stable for now.

October 20, 2007

Our new shell servers are now up and running. Please use to log instead of logging into calliope. If you need to use GSSAPI authentication with, you will need to follow the Kerberized SSH walkthrough to enable the GSSAPITrustDns option.

October 15, 2007

We have recently fixed the issue with Apache directory listings and believe that we have something very similar now up and running. The cluster is once again at full function, following unplanned downtime that occurred on October 13 due to power disruption.

October 13, 2007

We lost power and network to the cluster this morning due to some miscommunication with physical plant about the extent to which a power maintenance event would impact us. We are still scrambling to get everything up and working again, fingers crossed Update Services are up and running again. We have also added more shell servers to the to.ugcs rotation.

October 11, 2007

We will be doing a considerable amount of work on the cluster on Saturday, October 13. We do not expect any major downtimes; however, some login servers will be taken offline for short periods of time, and core servers may be rebooted causing brief interruptions. If you notice that something has been down for more than 5 minutes on Saturday, please contact us

October 3, 2007

All home directories were transferred as of Monday, October 1. Some files may have incorrect ownership since the copy was performed as root. If you have any files that were not transferred, or if you need to access the backups of UGCS3 files, please contact the sysadmins and we will copy your files from backup.

September 30, 2007

Home directories are now being copied into place; the process is about 30% complete but will run automatically (although a few users ran out of quota while I was copying and I'll need to reset their quotas and retry the transfer of their homedirs). There will be duplication of a lot of files - in particular, I've preserved your mail and www and cgi trees despite their having already been copied once before to a different location. To decrease your quota usage, you should delete any files you do not need immediately. Thanks for your patience!

September 28, 2007

All UGCS3 services have been shut down and transferred to UGCS4 servers, which are all now in the racks in the server closet. Home directories have been archived and will be shortly copied to the new cluster home directories. We do not anticipate any further major downtime.

If you still need to migrate your password, please use the link to the left. If you've forgotten your password, please e-mail - if you used SSH key login previously, we will put up your SSH key on a machine that will use your key to authenticate you and allow you to set a password on the new cluster.

Thanks very much for your patience, and we look forward to serving you.

Your UGCS Sysadmins

September 26, 2007

Home directories on UGCS3 will be frozen in the evening of September 27 in preparation for mass copying to the new fileservers. We are planning to bring down all services beginning as early as the afternoon of September 28 in order to rackmount all of the new hardware, carry out IP address reallocations, and reimage old core servers as user-accessible shell servers in UGCS4. The migration should be fully complete by the end of Saturday September 29.

September 22, 2007

Please make sure to read the Migration FAQs as they answer a number of topics related to the impact on user services during the course of the migration.

September 13, 2007

Hash: SHA1

Dear UGCS users,

We are proud to announce that we are about to complete the rollout of
UGCS 4.0, which will offer improved performance, features, and quotas.
However, these changes will require some action on your part, as well as
awareness that many of the quirks of UGCS behavior you are used to will
no longer be present.  If you have any questions, comments, or concerns,
please leave us an e-mail at and we'll
respond as soon as we can.

===Bottom line===
* We are targeting Saturday September 29 for the main switchover.  All
services will be unavailable on that day.
* Your password is frozen in its current state as of September 12.  You
will need to log into with
it to access mail, and to use the rest of the cluster when the migration
is complete.
* Your mail will only be accessible via secure IMAP and POP3, effective
very soon (tentatively September 16); you will need to verify your
password first as stated above.
* /ug/drop/mail is no longer writable, and all existent mailing lists
(except one-member lists) will be transferred as they are in their state
as of September 12.
* SSH keys will no longer work after the migration; we recommend use of
Kerberos for passwordless authentication.
* After the migration, your home directory will be copied to your new
home directory.  You will not be able to set per-file permissions, only
per-directory permissions.  The main portion of your home directory will
be not readable by anyone other than yourself; in order to share files
with other users, you will need to place them in the public subdirectory
of your home folder.
* If you wish to help us beta test the new system please send us an
e-mail and we will provide login instructions for our test machines.

We are migrating from NIS, which stores crypt() passwords, to Kerberos;
since crypt() is irreversible and Kerberos requires a copy of your
secret to create your principal, we cannot directly perform this
migration for you.  You will need to enter your old password and a new
password into an online form (using SSL). The application will then
enable your kerberos principal which you can subsequently use to access
all services on the cluster after the migration is done. Your migrated
password will be usable with mail (IMAP/POP3) immediately.  The
migration URL is the following:
The SHA1 fingerprint of the temporary self-signed certificate (until we
have time to properly establish a CA) is

===Network Filesystem===
We are migrating from NFS to AFS, a filesystem in wide use at other
universities including Stanford, MIT, and Carnegie Mellon.  AFS has
vastly improved security and speed compared to the version of NFS
currently in use on the cluster, not to mention better administrative
tools which will allow us to easily back up your data and move it
between servers to maximize performance.  AFS also allows user-settable
ACL's, eliminating the need to create custom groups for allowing subsets
of users access to data.  However, there are a few caveats: AFS does not
store permissions by file, only by directory.  We are defaulting to have
home directories remain readable only by their owners, with a
world-readable public subfolder.  If you wish to add a public file to
your home directory, place it in the public folder and symlink the
filename in your private home directory to the equivalent in your public
folder.  We have already set up a few such commonly-used symlinks on
your behalf such as .plan.  We will migrate your data for you from NFS
and place it in your home directory during the migration.

We have acquired approximately 3.2 terabytes of mass storage and 0.3
terabytes of fast SAS storage.  As a consequence, we are setting initial
quotas to 500 MB of mass storage for your home directory and 150 MB of
fast storage for your mail.  We reserve the right to modify these quotas
in the future, although they will most likely rise.  If you wish to have
a larger mail quota, please contact us - we can move your mail spool to
one of the mass storage machines and give you more space (at the penalty
of performance).

Your SSH keys will no longer function.  This is deliberate - AFS uses
Kerberos for authentication, which means that a Kerberos ticket is
required to mount your home directory; SSH keys cannot not provide
Kerberos authentication.  If you SSH to a machine directly and enter
your password, Kerberos tickets and AFS tokens are automatically
obtained for you using your password.  If you wish to use passwordless
authentication, we recommend that you install a Kerberos client on your
system and enable forwarding of tickets over SSH (GSSAPIAuthentication
and GSSAPIDelegateCredentials) for * in your
.ssh/config file if using *nix.

We are in the process of acquiring a number of new user-accessible Core
2 Duo systems, but all of the puke-class Pentium III machines will be
migrated for the present and the servers used for UGCS 3.0 services will
be decommissioned over time and integrated into the cluster as
user-accessible shell systems.

We have switched to using Maildir format for delivery of all new
messages.  Maildirs perform significantly better in a network filesystem
environment by avoiding the need to lock a single mbox file.  IMAP and
POP will only show messages from your Maildir.  We have used mb2md
( to place all the
messages from the mboxes we could identify in your Maildir.  If you wish
to manually migrate additional mboxes after the migration, you can
invoke mb2md yourself.  All inbound e-mail is now filtered using
amavisd, spamassassin, and clamav.  If you wish to forward your mail to
another address, you should update your LDAP entry with one (or
multiple) mailForwardingAddress entries instead of relying on .forward.
 Procmail is currently not in the mail delivery chain, but will be
integrated at a later date if it is still required by a large number of
users; we anticipate that the new mail stack will suffice for the
majority of users that were using procmail to invoke spamassassin or
perform filtering.  Additionally, since we are now able to filter all
inbound mail, we no longer need to greylist e-mails and therefore you
will no longer experience delays in delivery of mail to ugcs addresses.
 We have disabled non-secure IMAP and POP; you will need to use IMAP/S
or POP/S instead.  Like SSH, our IMAP and POP services are Kerberized
and you can authenticate without entering a password if you have a
Ticket-Granting-Ticket.  If you wish to send outbound e-mail using
UGCS's SMTP server, you also will need to authenticate either using your
password or a Kerberos ticket.

We are offering two new options for accessing your e-mail from a web
browser.  Roundcube is an AJAX webmail client that behaves like a
desktop mail client with drag and drop support.  Squirrelmail is more
traditional and works for the more paranoid about Javascript.  You can
go to or to access them.

===Mailing Lists===
We will be migrating all /ug/drop/mail lists to Mailman, a widely used
mailing list management tool that offers additional features such as
automatic removal of spammy messages, blocking of posts from
non-members, moderation, unsubscription, and archiving of messages.
Existing /ug/drop/mail lists have been frozen in preparation for the
migration.  We are offering a web-based list administration tool located
at in place of /ug/drop/mail.  For those
who use automated tools to manage /ug/drop/mail lists, please contact us
and we will advise you of the best way to handle automatic
additions/removals of list members.

===Public webhosting===
Your public_html folder will be automatically migrated and be served
from our new webserver.  We support PHP (version 5) and Perl through
SuExec.  By default, the web server will not be able to read files from
your home directory - if your website relies on files outside of the
public_html directory, they should by symlinked or moved into
~/public/public_html/. If you have questions about migrating your
existing web applications, please contact us.

Cluster machines will be running Debian testing (Lenny) with a set of
commonly used packages.  If you'd like to request a piece of software
which is currently not installed, please contact us and we'll add it to
the standard system image.  We hope that this central package management
will allow us to keep the software on UGCS as up-to-date as possible
with new versions and security updates.

===Database services===
Currently, database migration is not automated.  Please contact us to
get your database created and/or migrated.

In addition to continuing to support Gale, we are planning to set up a
Jabber server for your chatting convenience.

===Hosting and Authentication===
As the result of rearrangements made to our very limited pool of 62
usable IP addresses, we have needed to change the block of addresses
allocated to third-party hosting.  If you are hosting a server with us
and we have your contact information, we will send you your new
information to place in /etc/network/interfaces and will expect you to
configure your server appropriately or provide us with the access to
change the IP ourselves.  If not, you will have to track us down when
your server stops working.  In particular, there are a few bits you
_need_ to pay attention to with respect to specifying the correct MTU,
netmask, and routes.  Also, if your server remains offline for more than
a period of two weeks and we have no contact information on file for
you, we reserve the right to reallocate your IP to someone else.  If
your server is currently offline, we cannot automatically gather its MAC
address and will need this information from you if you wish to have an
allocation in the new network scheme.

As always, individuals in the Caltech community are welcome to colocate
servers with us.  We ask that you provide us with current contact
information in event we need to disrupt service to your server; we also
require your server's MAC address in order to place it on the
appropriate VLAN and provision you with a static IP.  We run network
intrusion detection software (Snort) to protect your server and also can
tighten firewall rules to restrict inbound traffic if you so desire.

Our Kerberos infrastructure is also available to others operating web or
other applications who need to validate the identity of a member of the
Caltech community.  Contact us for details if you are interested.


Your UGCS sysadmins (Elizabeth Fong, Matthew Maurer, Joshua Hutchins,
and Alex Roper)
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla -


August 22, 2007

Hash: SHA1

Dear UGCS users,

As the 20-year anniversary of UGCS in 2009 approaches, we are making
preparations for the next 20 years of UGCS in order to ensure that the
cluster is used by as many people as possible and continues to provide
top-notch services to the Caltech community. We are proud to announce
that we have been planning significant hardware and software upgrades to
UGCS over the course of the past six months which will result in vastly
improved performance, features, and quotas. In short, we are moving all
UGCS services to new, faster hardware and retooling the software
architecture to use commodity, well-supported software that we can
update and maintain in the coming years.

We hope to be finished with the initial migration by the beginning of
October. Please be advised that some UGCS services may need to be
temporarily disrupted during the buildout. Additionally, we may snapshot
the /ug/drop/mail system and the user password database for migration;
any changes following the snapshot will need to be reapplied after the
migration. A week before the migration, we will advise you of what
changes will impact you and any actions you may need to take. When we
switch over to the new infrastructure, we will need to bring down all
UGCS services for approximately one day.

If you have any questions, comments, or concerns, please send us an
e-mail at and we'll respond as soon as we can.


Your sysadmins (Elizabeth Fong, Matthew Maurer, Joshua Hutchins, and
Alex Roper)
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla -