-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

karin.namazu.org compromise report

We reports details of karin.namazu.org compromise. karin.namazu.org
was the main server of Namazu Project.

Time Table (JST)

5/19 Debian Project released new cvs package to fix security issue.

5/23 02:24 Occurred the first intrusion via CVS. The intruder sent
           some monitoring tools like ttymon.
     05:03 He sent some other tools a kind of keylogger.
     18:04-18:40 He created "test" user, and copied passwd and shadow
           files into his home directory.
     18:42 Exploited passwd and shadow file via ftp.
     19:07 The intruder was logined as the user "jitterbug" via ssh.
     19:56 Occurred a CVS access to escalate root privilege from
           jitterbug. The detail was unknown.
     19:58 He installed a kind of rootkit.
5/24 19:00 Network in the subnet of the place of karin.namazu.org was bursted.
     20:00 We find the compromise.
5/25 02:00 karin.namazu.org was unplugged from the network.

See http://www.namazu.org/#restoration for services restoration info.

Machines and services at the incident

CVS pserver was served as root privilege via inetd on karin.  The CVS
server provided many software and so many users can access via pserver
or ssh. Anonymous user also can get sources via pserver.

karin was build on Debian GNU/Linux 3.0. Latest cvs package was
released in 5/19, but we didn't upgrade it while the incident.

Then karin was unplugged by switching hub, and their HDD was moved
into another machine and analyzed.

In the result, we found some rootkit in the HDD. We decided 5/23 02:24
JST is the first intrusion time because some rootkit files and CVS
temporary directory had same ctime.

CVS pserver had root privilege, so the intruder could get root
privilege easily. And jitterbug account had temporary password to set
spam filter, and it was not removed, so he can get the password from
shadow file easily.

Recovering services

karin was old hardware, so we have a plan to move newer network and
machine named "vaj.namazu.org", and it was already placed. So we moved
all services into vaj.namazu.org.

Inspection CVS repository

karin had two HDDs, and one of them was used for backup. The original
CVS repository was daily backuped by rsync.  The original repository
was daily accessed to make ChangeLog graph, so almost files in the
repository had same atime.

On the other hand, rsynced backup repository files had correct atime
exclude directories. We considered the reason is that rsync accessed
only directory, then if any file was not changed, rsync didn't touch
the file.

It is possible to modify a file with keeping atime, but it requires to
record the atime before modify. We considered the possibility of
modify repository such complex sequences without inconsistency is very
low.  Then we checked difference between the backup and original, and
we can find only correct updates.

And we checked the release points version 2.0.12 and 2.0.13 from PGP
signed archives, and we can't find any difference.
Furthermore we checked the stable branch. A developer has a working
copy at 5/13, so he checked 5/13 stable branch and it and couldn't
find any difference, and checked further commits was correct.  We also
checked HEAD trunk with same method, and it seems no problem.

In the result, we considered the CVS repository is almost safe, and
continue to use it.

Further operations

Now we operate the following policies:

- - CVS pserver runs in chroot environment with non-root privilege, and
  it has the copy from original repository. It is for anonymous access
  only.
- -- The environment is built by Debian cvsd package. It is easy to
   update cvs command.
- - Reinforcement administration team. There is a mailing list for admin
  team, and the ml subscribe debian-security-announce list.
- - Non-admin members are only access CVS via ssh. It is a new
  accounting policy.
- - Make some backups from another network machines.

We are trying to operate more safety with the experience.

The analysis was cooperated with NetVillage Co., Ltd.

Jul 23, 2004
NOKUBI Takatsugu <knok@namazu.org>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBAKfMK6gmAsLOgJkRAlT4AJ9JhmXhvjhiMn0xCK2ib4EAo9Z0+QCgo8vM
bhItPf4G5H5p5ODLvORjbIs=
=Nvcp
-----END PGP SIGNATURE-----