| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When I use godo, error infomation comes along with other output, so
change all err report into stderr. And I listen to `pivot_root` sys-
call to find out the root file system of dockers. However, I'm afraid
of causing too more delay, so don't check rootfs of ppid and record
in the pid. Besides, the method to deal with pivot_root is hardcoded,
which may causes crush.
Shall I listen to the chdir syscall to find out exact cwd? Maybe It's
useful to the pivot_root?
Next step: Find out appropriate data stracture, and add more file
operations to be watched. This task must be completed this week.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
As previous envisioned, the lost is caused by slow consumption. So
I make several changes:
- Delete all the mutexs, especially those in the mongodb. There seems
to have no necessity to use thread mutex, because execve, fork
and exit has no conflicton(really?)
- Insert all pid info into db, just ws what we do to file infos. So
we should filter out useful info from them, but this does works
to decrease lost infos.
Besides, the problem that recvfrom is always blocked may got solved.
When the machine is just started, it's blocked; but after i run the
C program to connect to ketlink connector and listen to it, both C
program and godo can recv infos well.
Also, left questions:
- Now i use many coroutine in 2nd and 3rd coroutines in the hope that
there will be less time to deal info and hear the kernel again as
quick as possible. But does it work? As we know, too much thread
will slower the program, because too frequent switch between threads
or processes.
- Sometimes the eventTable has bugs, when eoe is recvd, the pointer
in it is null. It may be out of thread confliction? But it's
unreasonable, there's only one place to delete the event, that's
just in eoe, after it's sent. Or the eoe info is got more than once?
- For some processes, when i look into /proc to find cwd and cmeline,
it has exited. If i go back to use audit for pid info, it will be hard
to distinguish between thread and process. Anyway, It doesn't matter
now, but what if?
Next step: Figure out the root fs of a docker, and its name.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit I make several changes, reasons are as follows:
- Expand netlink connector buffer size. Originally it uses only one
page of mem, but sometimes it causes "no enough buffer" errno, or
the socket is blocked long time. Now it's 20 pages.
- All file infos are thrown into database. As the last commit co-
mment, There's 2 tables, "fds" and "files". When a file discriptor
is closed, the info in fds will be found, delete, and put into the
"file" table with its close time.
Left questions:
- The netlink connector is always found blocked without any reasons.
Fix it, or replace the golang-coded connector with C program? The
key is why it's blocked. Maybe it's in the kernel src code.
- sometimes audit still losts info(not too much). For instance, I
use vim in the docker to change hello.c, the hello.c may be opened
but no close info recvd. Or, the swap file of vim, such as
.hello.c.swp or .hello.c.swx is not closed. What's more, the hello.c
is never written, but swap files are. May vim write to swap files,
and replace the origin file? Let's check it.
- Besides, when a pid exits, we should check its file discriptors
and close them all.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There are some possible reasons that have been thought:
- auditd lost. Each time I use `auditctl -b xxx` or `auditctl
--reset-lost`, there are always a big number of losts. at first i
thought it means how many auditd info was lost throw the net, or
how many was thrown because of the audit info queue in the kernel
was full. However, form the src code of kernel, it actually means
how much is thrown away as there's no listener of auditd info. In
other words, audit is a userspace-kernel function, but not two
independent parts.
- audit backlog size. As the above.
But when i only listen to the syscall "open", i can almost always
hear the info in the docker. So I think this may be because the
audit info production is flooding, while in this program i check this
and that, causes too much time, the consumption is far slower.
Next step, I will use the MVC, all recvd info will be push into the
database, and add a new independent part to make database clean and
clear.
The key problem is, a process can open file1 as fd 3, write, close,
and open file2 as fd 3, write, close: which means i must figure out
which file to write when "write" event comes. Now i check the
pid/fd/close_time in database to choose which is written, but find
and check doc also use lots of time. Maybe, use two collections, one
is fds that records files not closed, the other records closed files?
Besides, as clone/fork/pthread_create all uses syscall clone, but
their flags are different. Maybe i can also use `pid/tgid` pair to
distinguish between process and thread. Good idea.
Be quick, your internship has passed a half. What kinds of answer
will you hand in?
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It's the check(cooked Event) function that causes fileopen crushed,
and now I'am sorry to say that i've forgot why i add this function,
maybe to check ppid and pid in database in just one function but not
the head of each function. However, the check in each function isn't
deleted. I discover it by comparing source code with 5d244e3. In
theory this would only result in the increase of delay. How does it
affect on the fileopen and causes failure? No one knows.
The same to kernel connector. If we still add delay while pid exits,
the connector will say "Error recv: no enough buffer space", but if
we delete the delay, all modules work well. What actually makes the
delay in pid exit causes no enouth buffer of connector? How outra-
geous it is!
Now I've come back to the original question: when i start and use
docker quickly(`start && exec && exit` in just one command), the file
open/write/close is faithfully recorded; but if i use interactive
shell and use vim to change file in docker, nothing happens.
Why? Why? Why?
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For some reasons, kernel-connector can catch exec event, but it
doesn't tell me about what the process exec and what're its args.
So we should use audit to collect these infomations, and complete
in the database.
However, there's different delays between connector and audit,
although they both use netlink socket, as a result of which, exec
may comes before fork. we deal with it the same way. But, there's
also exec event lost, may because of the check for ppid in exec
event, but it's necessary, and if is deleted, too much irrelavent
infomation would flood into database, i've tried. So make it there,
just go forward.
Besides, what's newly discovered is that pthread_create also use
clone syscall, but if pid 1 has a thread 2, the exec info will say
that pid 2 execs. So i shouldn't ignore connector msg that childPid
ne childTgid.
This is my first attempt to use git-submodule function in my own pro-
ject, also golang local package. Congratulations!
Now, fight to fix about file operations. Hope that there wouldn't
be too many fucking bugs.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
| |
this commit i successfully catch open/close syscall, and insert them
as an independent collection in mongodb otherwise along with pids.
and now I've record those open flag "O_TRUNC" as written.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There's 2 bugs from ancestor commits:
- In the 'things_left' tag commit(the grandpa of this commit), we
add a function that allows execve comes before fork, but when it
happens, I forget to insert the basic info (pid, ppid, etc.), as a
result of which it doesn't work in the designed way. Now it is well,
insert execve with pid and ppid, so that the fork event can find it
and finish other info. However, we shouldn't make start_stamp in
this case, so that it's also a flag. I've not removed the unused
execve info, waiting for the future.
- In the parent commit, the syscallRegex is changed, because when we
add more syscalls to be watched, we need more info about their params
but not only the first one. Instead of keeping using single a0 to get
the first param, i use argsRegex for all the params. But this change
causes mismatch of syscallRegex. Now it's fixed.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
To record it, we must listen to open/write and several syscalls,
and now I've add open into the 2nd coroutine. In syscall open,
what we should do is to judge the permission flag (the 2nd param
in the syscall), to find out if it can write to the file. If so,
the exit code is its file descriptor, and when write is called, the
audit shows only file descriptor but no file name.
So the next step is to add things into 3rd coroutine, to make the
whole program running again, and find out bugs.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The Most important work during this time is to find out solution
to the out-of-order bug. Discribe it here in detail: info from
audit may be out of order, which means fork may comes after execve,
even after exit. What an absurd penomenon to see a process not yet
created to work or exit!
To deal with this problem, I've tried several ways:
- in the 2nd coroutine, when EOE msg comes, if it's a fork/clone
event, send it immediately, otherwise wait for some time(such as
100 ms). But after all it delays longer, and has other problems.
- the 2nd coroutine doesn't send directly, but record all the finished
event id in a slice, and another thread checks once every one second,
if there are sth in slice, send corresponding events in the order of
event id. But: event that happens first doesn't always has lower id
or time, for example, 1 forks 2, then 2 execve, the audit in kernel
it self may gets execve before fork(maybe fork makes other settings),
which means execve has earlier timestamp and lower event id. The out-
of-order problem is not completely resolved. If we then add delays
to non-clone event, a more serious problem happens: we must use mutex
to lock the slice recording finished event id to prevent crush between
send thread and wait thread, but the wait thread can't get the mutex
again, because there are to much clone event and frequent send!
- So I use no delay but mongodb, when an execve comes, if pid is not
recorded, just insert it and wait for the fork. It does works, but
some other works is still left to do:
- what should i do if 2 forks 3 comes before 1 forks 2? Now I
suggest it doesn't happen, but what if?
- when execve comes before fork, i recorded it, but if this process
has a parent i don't care, delete, or stays there?
Also, as mentioned above, I've add EXECVE field in process into db,
records all the execve(time, and args) from the same process. Besides,
exit_timestamp and exit_code can be caught now, but too many process
has no exit info. This is also to be fixed.
Now, let's listen to the file changed by process. Don't forget the
to-do works listed above!
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I failed to print the process tree out. While I'm printing the tree,
the tree itself gets changed, maybe deleted. What's more, the output
show that there are 4 lines with the same ppid and pid, how an absurd
result! It may be caused by multi-thread. So, use database instead.
Mongodb uses bson(binary json) to store data but not relational
database like mysql, which means it's more easy to use.(?)
Beside inserting, I've also solved a question that "fork" is called
once but returns twice. For instance, pid 1 forked pid 2, in the
audit log it's not an event "syscall=clone,ppid=1,pid=2", but actually
two events "syscall=clone,exit=0,ppid=0,pid=1" and "syscall=clone,exit=
2,ppid=0,pid=1", which is just what we see in sys_fork in kernel source.
To deal with this, when syscall is clone and exit is 0 we just drop it.
Left question: To find out the exit code when a process exit/exit_group,
and finish the code to record it in the database.
|
| |
|
|
|
| |
Put all the src code in only one file is to ugly, so devide it!
and mv them into src dir to keep the whole repo clear.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For some reasons, the linux kernel has made some changes in syscalls.
As shown in src code, we pay attention to fork/vfork/clone to create
process, while exit/exit_group to kill it. From my opinion, the fork
and clone syscall should be totally different, otherwise there will
be only one syscall. However, according to the logs, I heard only
clone but no fork, exit_group but no exit. Infact, fork calls clone
and then makes some special set. They're different, fork means parents
and children, while clone means calling and callee, which allows to
share sth between caller and callee. Both fork and clone makes a new
process, pthread makes tasks and is called thread. Pid is factually
task id.
Now the 3 coroutines works well, and I've get a process tree by
map[int]*process. Here hides some questions:
- is it right for 2nd corutine to send to 3rd as long as eoe?
- how to make the delay between exit_group and deletePid clear and
suitable?
Next works:
- Change the pids from map into DataBase, which means that we
should devide front-end and back-end. Besides, when you delete
sth(such as process exit), don't delete from databese, instead
just make a tag and record their exit code. In other words, we
judge if it's alive not by entry existance but exit tag.
- Make containers recorded, for instance, rootFS, root-process,
name, id, etc.. And record them in map, maintain this database table.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
As is planed, the first coroutine throw rae event infomation to the
second, and it organizes all info for the same event accroding to
event id, which is unique without shutdown of this computer.
There's several defficuties I've encountered, so I list their solution
here to remeber:
- raw info from 1st coroutine is correct, but wrong when 2nd gets it;
or it's correct while recieved, then regular expr goes to match it,
the first match is inline with expectations, but the next match goes
totally wrong, and the info is different from what is received.
Look into the src of go-libaudit, we'll find out that when heard
from netlink socket, the read buffer is always the same slice, it
first received a long data, then **pass the origin slice to
rawEvent.Data**, and then received a shorter data. rawEvent.Data is
passed to 2nd coruntine as **a pointer to rawEvent**, which means
all this 3 process use the same part of memory. Then, when a shorter
info comes from socket, the slice won't be moved, otherwise it write
aigin to this part of mem, then coroutine 2 will get a dirty data.
To deal with it, we change the type of channel from pointer to
interface, and make a deep copy of rawEvent before passing down.
As a result, the 2nd coroutine gets a copy of message but not origin,
it finally comes right.
- While designing a regular expr, it's thought correct but miss
matched from the right string. There maybe sth wrong that can't be
discovered by people's eye, you can try to rewrite the expr, then
it may be fixed.
Also, there's some hidden dangers:
- 2nd coroutine comes with no error checks althouth err variable is
set and catched ubder the rules of compiler. we **shall** make it
later.
- Is it reasonable to pass cooked event info immediately to 3rd
coroutine without waiting some time? Info from network is out of
order after all.
Fight! Fight! Fight!
|
|
|
This repo is to supervise all processes in containers, in other
words inspect behaviors of dockers, and get the pid tree.
There are several ways for programs in user space to intereact with
kernel space:
- system calls, which can be found out in source path arch/x86/syscalls
- ioctl
- /proc virtual file system, to read kernel realtime info
- nerlink socket
the pid we should pay attention to is /usr/bin/containerd, which may
come from service docker-daemon and ppid is 1. Each time a docker is
start or stop, this forks a pid, the pid then forks, that's the main
process of the docker.
To grub the info of pid create or exit, this program is based on
go-libauditd, which uses netlink socket to hear from kernel about
audit log. What's worrying is that one event is always devided into
several entries, and several events may be received alternately.
So, from my point of view, which program has 3 coroutines and 2
channels. the first receives raw event message from audit, then
throw it to channel 1; the second listen to channel 1, and organizes
each event until there's a EOE, then throw to channel 2; the third
discover event from channel 2, deal with th event, such as create or
delete pid. Specially, since two relative infomation(pid 1 fork pid2,
then pid 1 exits)may comes out of order, deletion mast be delayed for
some time(may 1 second), to keep the process tree correct.
|
We-unite