OcIc@sdZddklZddkZddkZddkZddkZddkZddkZddk Z ddk Z ddk Z ddk Z ddk Z ddkZdefdYZdZdZdZeid eiZd Zeid Zd Zd ZdZdddYZdefdYZdddYZdefdYZdd dYZ dd!dYZ!de!fdYZ"da$dZ%ddk&Z&ddk'Z'dS("saStore and retrieve metadata in destination directory The plan is to store metadata information for all files in the destination directory in a special metadata file. There are two reasons for this: 1) The filesystem of the mirror directory may not be able to handle types of metadata that the source filesystem can. For instance, rdiff-backup may not have root access on the destination side, so cannot set uid/gid. Or the source side may have ACLs and the destination side doesn't. Hopefully every file system can store binary data. Storing metadata separately allows us to back up anything (ok, maybe strange filenames are still a problem). 2) Metadata can be more quickly read from a file than it can by traversing the mirror directory over and over again. In many cases most of rdiff-backup's time is spent compaing metadata (like file size and modtime), trying to find differences. Reading this data sequentially from a file is significantly less taxing than listing directories and statting files all over the mirror directory. The metadata is stored in a text file, which is a bunch of records concatenated together. Each record has the format: File ... Where the lines are separated by newlines. See the code below for the field names and values. i(t generatorsNt ParsingErrorcBseZdZRS(s6This is raised when bad or unparsable data is received(t__name__t __module__t__doc__(((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyR=scCs|pdSng}|idti|d|idti|d|id|d|id|d y|id |d Wn#tj otid d nXdi|S(s9Convert CarbonFile data to a string suitable for storing.tNones creator:%stcreatorstype:%sttypeslocation:%d,%dtlocationsflags:%dtflagss createDate:%dt createDates7Writing pre-1.1.6 style metadata, without creation datei t|(tappendtbinasciithexlifytKeyErrortlogtLogtjoin(tcfilet retvalparts((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pytcarbonfile2stringAscCsh}x|idD]}|id\}}|djoti||dThis is called on the match obj of any char that needs quotingis s\ns\s\\sBad char %s needs quotingN(tgroupR>(t match_objtchar((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pytreplacement_funcs   (tchars_to_quotetsub(t path_stringR((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyR+s cCsd}tid||S(s#Reverse what was done by quote_pathcSsQ|id}|djodSn|djodSntid|d|S(s+Unquote match obj of two character sequenceis\ns s\\s\s)Warning, unknown quoted sequence %s foundi(RRR(Rt two_chars((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRs  s\\n|\\\\(treR(t quoted_stringR((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRys cCs2|djodSntt|idSdS(s(Return tuple index given quoted filenamet.t/N((ttupleRyR(tquoted_filename((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRws t FlatExtractorcBsYeZdZdZdZdZdZdZdZ dZ dZ dZ RS( s)Controls iterating objects from flat filecCs(||_d|_d|_d|_dS(NR$ii ii(tfileobjtbuftat_endt blocksize(tselfR((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyt__init__s   cCsxx|ii|id}|o|idSq|ii|i}|pd|_t|iSq|i|7_qdS(s<Return position of next record in buffer, or end pos if noneiN( trecord_boundary_regexptsearchRtstartRtreadRRtlen(Rtmtnewbuf((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyt get_next_poss ccsqxj|iD]\}y|i|VWq ttfj o/}|ioPntid|fdq Xq WdS(s4Return iterator that yields all objects with recordssError parsing flat file: %siN(titerate_recordstrecord_to_objectRt ValueErrorRRR(Rtrecordte((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pytiterates  ccspxQ|i}|io|o|i| VnPn|i| V|i||_q|ii ptdS(sYield all text records in orderN(RRRRtcloseR>(Rtnext_pos((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRs   cCs|i p|iidptx|ii|i|_|i|ii7_|ipd|_dSnx|ii |i}|pPn|i |i d}||jo!|i|i d|_dSqv|i|i d|_qvq(dS(sScan through the file, set buffer to beginning of index record Here we make sure that the buffer always ends in a newline, so we will not be splitting lines in half. s iNi(RtendswithR>RRRtreadlineRRRtfilename_to_indexRRtend(RRRt cur_index((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyt skip_to_index#s %   ccs|i||iodSnx|i}y|i|i| }Wn2ttfj o }tid|fdn&X|i t | |joPn|V|ioPn|i||_q"|i i pt dS(s3Iterate objects whose index starts with given indexNsError parsing metadata file: %si(RRRRRRRRRRRRRR>(RRRtobjR((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pytiterate_starting_with:s     cCsdptdS(sTranslate filename, possibly quoted, into an index tuple The filename is the first group matched by regexp_boundary_regexp. iN(R>(Rtfilename((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRJsN( RRRRRRRRRRRRR(((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRs    t RorpExtractorcBs5eZdZeidZeeZee Z RS(s Iterate rorps from metadata files(?:\n|^)(File (.*?))\n( RRRRtcompileRt staticmethodRRRwR(((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRTs tFlatFilecBseZdZd \ZZZdZd \ZZ e Z d Z d Z ddd dZdZdZd dZdZdZRS( sManage a flat (probably text) file containing info on various files This is used for metadata information, and possibly EAs and ACLs. The main read interface is as an iterator. The storage format is a flat, probably compressed file, so random access is not recommended. iidcs)|_|_g_|o7|io|iijp t|d}n|djo%|_iid|_ n|djpt|o=|o6|i o(fd}t i ||_ nC|_ii p tiiidd|_ dS( sOpen rp (or rp+'.gz') for reading ('r') or writing ('w') If callback is available, it will be called on the rp upon closing (because the rp may not be known in advance). itrtrbtwcs |_dS(N(trp(R(R(s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pytcallbackstwbtcompressN(tmodeRt_record_buffert isincfiletgetincbase_strt_prefixR>RtopenRtisinccompressedRzt MaybeGziptlstat(Rtrp_baseRt check_pathRR((Rs:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRjs"         cCst|ioV|ii|t|i|ijo)|iidi|ig|_qpn|ii|dS(s#Write a (text) record into the fileR$N(t _buffering_onRR Rt_max_buffer_sizeRtwriteR(RR((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyt write_records  cCs|i|i|dS(s.Convert one object to record and write to fileN(Rt_object_to_record(Rtobject((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyt write_objectscCs@|p|i|iiSn|i|i}|i|S(s/Return iterator of objects records from file rp(t _extractorRRR(Rtrestrict_indext extractor((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyt get_objectsscCs|i|iiS(sReturn iterator of text records(RRR(R((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyt get_recordsscCs|ip td|io3|io)|iidi|ig|_n|ii}d|_|ii |ii |i o|i |in|S(s(Close file, for when any writing is donesFile already closedR$N( RR>RRRRRRRtfsync_with_dirtsetdataR(Rtresult((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRs     N(NNN(Nid(RRRRRRRRRRRRRRRRRRRR(((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyR[s     t MetadataFilecBs&eZdZdZeZeeZRS(s5Store/retrieve metadata from mirror_metadata as rorpstmirror_metadata( RRRRRRRRQR(((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRstCombinedWritercBs)eZdZdZdZdZRS(s8Used for simultaneously writting metadata, eas, and aclscCs*||_||||_|_|_dS(N(t metawriterteawritert aclwritert winaclwriter(RRRRR((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRs cCs|ii||io.|ii o|ii|in|io.|ii o|ii|in|io|ii|i ndS(s,Write information in rorp to all the writersN( RRRtget_eatemptyRtget_acltis_basicRt get_win_acl(Rtrorp((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRs cCsb|ii|io|iin|io|iin|io|iindS(N(RRRRR(R((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRs    (RRRRRR(((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRs  tManagercBseZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z dd Zd ZdZdZdZdZdddZRS(s)Read/Combine/Write metadata files by timeRtaccess_control_liststextended_attributestwin_access_control_listscCsjg|_hh|_|_xGtiiD]6}tii|}|io|i|q,q,WdS(s$Set listing of rdiff-backup-data dirN( trplistt timerpmapt prefixmapR3trbdirtlistdirR Rt add_incrp(RRR((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRs  cCs|ip t||ii||i}|ii|o|i|i|n|g|i|<|i}|ii|o|i|i|n|g|i|RR t getinctimeRthas_keyRR(RRttimetincbase((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRs  cCsd|ii|pdSnxB|i|D]3}|i|jo||di|Sq)q)WdS(s1Used below to find the right kind of file by timeRN(RRRRR(Rtprefixt flatfileclassRRR((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyt _iter_helperscCs|i|it||S(s5Return iter of metadata rorps at given time (or None)(Rt meta_prefixR(RRR((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pytget_meta_at_timescCs|i|iti||S(s7Return Extended Attributes iter at given time (or None)(Rt ea_prefixteas_aclstExtendedAttributesFile(RRR((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pytget_eas_at_times cCs|i|iti||S(s8Return ACLs iter at given time from recordfile (or None)(Rt acl_prefixRtAccessControlListFile(RRR((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pytget_acls_at_times cCs|i|iti||S(s9Return WACLs iter at given time from recordfile (or None)(Rt wacl_prefixtwin_aclstWinAccessControlListFile(RRR((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pytget_win_acls_at_times cCs@|i||}|ptidddSntioO|i||}|p tiddtg}nti ||}nti oO|i ||}|p tiddtg}nti ||}nti oO|i||}|p tiddtg}nti||}n|S(s;Return combined metadata iter with ea/acl info if necessarys\Warning, could not find mirror_metadata file. Metadata will be read from filesystem instead.is+Warning: Access Control List file not founds+Warning: Extended Attributes file not founds4Warning: Windows Access Control List file not found.N(RRRRR3t acls_activeRtiterRt join_acl_itert eas_activeRt join_ea_itertwin_acls_activeRRtjoin_wacl_iter(RRRtcur_itertacl_itertea_itert wacl_iter((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyt GetAtTimes2     cCs|djo ti}nti|}d|||f}tii|}|i ptd|i f|i pt||dd|i S(s;Used in the get_xx_writer functions, returns a writer classs%s.%s.%ssFile %s already exists!RRN( RtTimet curtimestrt timetostringR3RR RR>tpathRR(RRRttypestrRttimestrRR((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyt_writer_helpers "cCs|i|it||S(s;Return MetadataFile object opened for writing at given time(RRR(RRR((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pytget_meta_writer)scCs|i|iti||S(s0Return ExtendedAttributesFile opened for writing(RRRR(RRR((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyt get_ea_writer.s cCs|i|iti||S(s/Return AccessControlListFile opened for writing(RRRR(RRR((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pytget_acl_writer3s cCs|i|iti||S(s2Return WinAccessControlListFile opened for writing(RRRR(RRR((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pytget_win_acl_writer8s tsnapshotcCs|i||}ti oti oti o|Sntio|i||}nd}tio|i||}nd}tio|i||}nd}t ||||S(s=Get a writer object that can write meta and possibly acls/easN( RR3RRRRRRRR(RRRRt ea_writert acl_writertwin_acl_writer((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyt GetWriter=s    N(RRRRRRRRRRRRRRRR RRRRRR(((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRs&           t PatchDiffMancBsVeZdZdZdZddZdZdZdZdZ d Z RS( sContains functions for patching and diffing metadata To save space, we can record a full list of only the most recent metadata, using the normal rdiff-backup reverse increment strategy. Instead of using librsync to compute diffs, though, we use our own technique so that the diff files are still hand-editable. A mirror_metadata diff has the same format as a mirror_metadata snapshot. If the record for an index is missing from the diff, it indicates no change from the original. If it is present it replaces the mirror_metadata entry, unless it has Type None, which indicates the record should be deleted from the original. i ccsgx`ti||D]L\}}|pti|iVq| p|i|ijo |VqqWdS(s*Iterate meta diffs of new_iter -> old_iterN(trorpitert Collate2ItersRzR{RR(Rtnew_itertold_itertnew_rorptold_rorp((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyt get_diffiter_s  icCs|ii|pgSng}|i|D]}||i|fq-~}|i|ig}|D]$\}}||jo ||qnqn~S(s>Return reverse sorted (by time) list of incs with given prefix(RRRtsorttreverse(RRtmin_timet_[1]Rtsortlistt_[2]R((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pytsorted_prefix_inclistgs 4  cCs|id}t|djptt|djodSn|d \}}|i|ijo djnptd}x4|dD](}|idjoPn|d7}qW||ijodSn||fS( s@Check if we should diff, returns (new, old) rps, or (None, None)RiiRtdiffN(NN(NN(R+RR>Rt getinctypetmax_diff_chain(RtinclisttnewrptoldrptchainlenR((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pytcheck_needs_diffos1 cCs|i\}}|pdSntidd|id|i}t|di}t|di}x'|i||D]}|i|qW|i |i dS(s9Replace a mirror snapshot with a diff if it's appropriateNsWriting mirror_metadata diffiR,R( R3RRRRRRR$RRtdelete(RR0R1t diff_writerR R!t diff_rorp((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pytConvertMetaToDiff~s cCswg}|i|D]}|t|di|q~}|pdSnt|djo |dSn|i|S(s7Get metadata rorp iter, possibly by patching with diffsRiiN(trelevant_meta_incsRRRRtiterate_patched_meta(RRRR(Rt meta_iters((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRs5 cCs|idd|}|p|Sn|di|jpt|dxGtt|dddD])}||idjo ||SqfqfWdptd|fdS( s1Return list [snapshotrp, diffrps ...] time sortedRR'iiRis Inclist %s contains no snapshotsN(R+RR>trangeRR-(RRR/ti((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyR8s%ccsx}ti|D]l}xctt|dddD]4}||o#||io ||VnPq3q3Wdp tdqWdS(sReturn an iter of metadata rorps by combining the given iters The iters should be given as a list/tuple in reverse chronological order. The earliest rorp in each iter will supercede all the later ones. iiisNo valid rorpsN(RtCollateIteratorsR;RRR>(Rtmeta_iter_listt meta_tupleR<((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyR9s   ( RRRR.R$R+R3R7RR8R9(((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyRMs      cCs tatS(N(Rt ManagerObj(((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pyt SetManagers ((((((Rt __future__RRtgziptosR RR3RzR trobustt incrementtstaticRt ExceptionRRR"RQRtMRuRRR+RyRwRRRRRRRRR@RARR(((s:/var/lib/python-support/python2.5/rdiff_backup/metadata.pys7s.0`  D ,  bO`