v0.8
/- osd recovery fixes
/- backup homiebackup5
/- backup backup.
/- bug fixes?
/- mount error reporting
/- fix mds file size/mtime recovery
/- changes for 2.6.30
/- mds cap fixes
/- crush fixes

v0.9
/- make mds exhert memory pressure on client caps, leases
/- librados
/  - async io
/  - list_objects
/  - rados
/    - --snap N
/    - fetch/store/remove object
/    - list objects
/    - list pools
/- object classes
/- crush bugfixes
/- kclient readdir fixes

bugs

osd/OSD.cc: In function 'void OSD::finish_recovery_op(PG*, int, bool)':
osd/OSD.cc:3462: FAILED assert(recovery_ops_active >= 0)
 1: ./cosd(_ZN3OSD18finish_recovery_opEP2PGib+0x34f) [0x4a6f9f]
 2: ./cosd(_ZN12ReplicatedPG17sub_op_push_replyEP14MOSDSubOpReply+0x45d) [0x46681d]
 3: ./cosd(_ZN3OSD10dequeue_opEP2PG+0x24b) [0x4acc6b]
 4: ./cosd(_ZN10ThreadPool6workerEv+0x1b4) [0x56c204]
 5: ./cosd(_ZN10ThreadPool10WorkThread5entryEv+0xd) [0x4d476d]
 6: ./cosd(_ZN6Thread11_entry_funcEPv+0xc) [0x45f2ac]
 7: /lib/libpthread.so.0 [0x7fdb66e293f7]
 8: /lib/libc.so.6(clone+0x6d) [0x7fdb663f894d]

mds/Server.cc: In function 'void Server::_finish_session_purge(Session*)':
mds/Server.cc:410: FAILED assert(session->is_stale_purging())
 1: ./cmds(_ZN6Server21_finish_session_purgeEP7Session+0x392) [0x49edf2]
 2: ./cmds(_ZN6Server18find_idle_sessionsEv+0xa18) [0x4a3188]
 3: ./cmds(_ZN3MDS4tickEv+0x220) [0x484f60]
 4: ./cmds(_ZN9SafeTimer12EventWrapper6finishEi+0x1c1) [0x63eb11]
 5: ./cmds(_ZN5Timer11timer_entryEv+0x6f6) [0x6412d6]
 6: ./cmds(_ZN5Timer11TimerThread5entryEv+0xd) [0x46d53d]
 7: ./cmds(_ZN6Thread11_entry_funcEPv+0xc) [0x480c9c]
 8: /lib/libpthread.so.0 [0x7f51a9f4c3f7]
 9: /lib/libc.so.6(clone+0x6d) [0x7f51a951b94d]


later
- client reconnect after long eviction; and slow delayed reconnect
- ENOSPC
  - space reservation in ObjectStore, redeemed by Transactions?
  - reserved as PG goes active; reservation canceled when pg goes inactive
  - something similar during recovery
  - ?
- repair
- mds security enforcement
- client, user authentication
- cas
- osd failure declarations
- rename over old files should flush data, or revert back to old contents

rados
- perl swig wrapper
- call method
- merge pgs
- destroy pg_pools
- autosize pg_pools?
- security

repair
- repair metadata.. 
/  - parent pointer on CDir objects
  - update on dir rename commit
- namespace reconstruction tool
- repair pg (rebuild log)  (online or offline?  ./cosd --repair_pg 1.ef?)
- repair file ioctl?
- are we concerned about
  - scrubbing
  - reconstruction after loss of subset of cdirs
  - reconstruction after loss of md log
- data object 
  - path backpointers?
  - parent dir pointer?
- cdir objects
  - parent dir pointer
    - update on rename?  or on cdir store?
      on cdir store is sufficient if mdlog survives...
  - or what the hell, full trace?
- mds scrubbing

kernel client
- fix up mds selection, and ESTALE handling
- make cap import/export efficient
- simplify mds auth tracking?
  - use caps instead?
- unwind writeback start error in addr.c (see fixme)... by redirtying pages?
- flock, fnctl locks
- ACLs
- make writepages maybe skip pages with errors?
  - EIO, or ENOSPC?
  - ... writeback vs ENOSPC vs flush vs close()... hrm...
- set mapping bits for ENOSPC, EIO?
- should we try to ref CAP_PIN on special inodes that are open?  
- fix readdir vs fragment race by keeping a separate frag pos, and ignoring dentries below it
- reconnect after being disconnected from the mds
- inotify for updates from other clients?

vfs issues
- real_lookup() race:
  1- hash lookup find no dentry
  2- real_lookup() takes dir i_mutex, but then finds a dentry
  3- drops mutex, then calld d_revalidate.  if that fails, we return ENOENT (instead of looping?)
- vfs_rename_dir()
- a getattr mask would be really nice

filestore
- make min sync interval self-tuning (ala xfs, ext3?)
- get file csum?

btrfs
- clone compressed inline extents
- ioctl to pull out data csum?


userspace client
- handle session STALE
- time out caps, wake up waiters on renewal
  - link caps with mds session
- validate dn leases
- fix lease validation to check session ttl
- clean up ll_ interface, now that we have leases!
- clean up client mds session vs mdsmap behavior?
- stop using mds's inode_t?
- fix readdir vs fragment race by keeping a separate frag pos, and ignoring dentries below it

mds
- add an up:shadow mode?
  - tail the mds log as it is written
  - periodically check head so that we trim, too
- handle slow client reconnect (i.e. after mds has gone active)
- anchor_destroy needs to xlock linklock.. which means it needs a Mutation wrapper?
  - ... when it gets a caller.. someday..
- add FILE_CAP_EXTEND capability bit
- dir fragment
  - maybe just take dftlock for now, to keep it simple.
- dir merge
- snap
  - hard link backpointers
    - anchor source dir
    - build snaprealm for any hardlinked file
    - include snaps for all (primary+remote) parents
  - how do we properly clean up inodes when doing a snap purge?
    - when they are mid-recover?  see 136470cf7ca876febf68a2b0610fa3bb77ad3532
  - what if a recovery is queued, or in progress, and the inode is then cowed?  can that happen?  
  - proper handling of cache expire messages during rejoin phase?
    -> i think cache expires are fine; the rejoin_ack handler just has to behave if rejoining items go missing

- clustered
  - on replay, but dirty scatter replicas on lists so that they get flushed?  or does rejoin handle that?
  - linkage vs cdentry replicas and remote rename....
  - rename: importing inode... also journal imported client map?



journaler
- fix up for large events (e.g. imports)
- should we pad with zeros to avoid splitting individual entries?
  - make it a g_conf flag?
  - have to fix reader to skip over zeros (either <4 bytes for size, or zeroed sizes)
- need to truncate at detected (valid) write_pos to clear out any other partial trailing writes


mon
- paxos need to clean up old states.
  - default: simple max of (state count, min age), so that we have at least N hours of history, say?
  - osd map: trim only old maps < oldest "in" osd up_from

osdmon
- monitor needs to monitor some osds...

pgmon
/- include osd vector with pg state
  - check for orphan pgs
- monitor pg states, notify on out?
- watch osd utilization; adjust overload in cluster map

crush
- allow forcefeed for more complicated rule structures.  (e.g. make force_stack a list< set<int> >)

osd
- preserve pg logs on disk for longer period
- optionally separate osd interfaces (ips) for clients and osds (replication, peering, etc.)
- pg repair
- pg split should be a work queue
- generalize ack semantics?  or just change ack from memory to journal?  memory/journal/disk...
- rdlocks
- optimize remove wrt recovery pushes

simplemessenger
- close idle connections?

objectcacher
- read locks?
- maintain more explicit inode grouping instead of wonky hashes

cas
- chunking.  see TTTD in
   ESHGHI, K.
   A framework for analyzing and improving content-based chunking algorithms.
   Tech. Rep. HPL-2005-30(R.1), Hewlett Packard Laboratories, Palo Alto, 2005. 
