In some locales the default encoding is ascii in which case
subprocess.check_output() will fail if it is given a non-ascii ref as
one of the arguments. By forcing the ref to be utf8 we will avoid a
crash while still behaving correctly when the default encoding is
utf8.
The credits for this fix go to Nikita Bazhinov for discovering the fix
and Chris J Billington for explaining it.
Co-Authored-By: Nikita Bazhinov <nbazhinov@syntellect.ru>
Co-Authored-By: Chris J Billington <chrisjbillington@gmail.com>
hg-fast-export.sanitize_name expects branch name to be a bytes
object. Command line parser gives out str objects. Convert
possible str object to bytes in hg2git.set_default_branch().
Port hg-fast-import to Python 2/3 polyglot code.
Since mercurial accepts and returns bytestrings for all repository data,
the approach I've taken here is to use bytestrings throughout the
hg-fast-import code. All strings pertaining to repository data are
bytestrings. This means the code is using the same string datatype for
this data on Python 3 as it did (and still does) on Python 2.
Repository data coming from subprocess calls to git, or read from files,
is also left as the bytestrings either returned from
subprocess.check_output or as read from the file in 'rb' mode.
Regexes and string literals that are used with repository data have
all had a b'' prefix added.
When repository data is used in error/warning messages, it is decoded
with the UTF8 codec for printing.
With this patch, hg-fast-export.py writes binary output to
sys.stdout.buffer on Python 3 - on Python 2 this doesn't exist and it
still uses sys.stdout.
The only strings that are left as "native" strings and not coerced to
bytestrings are filepaths passed in on the command line, and dictionary
keys for internal data structures used by hg-fast-import.py, that do
not originate in repository data.
Mapping files are read in 'rb' mode, and thus bytestrings are read from
them. When an encoding is given, their contents are decoded with that
encoding, but then immediately encoded again with UTF8 and they are
returned as the resulting bytestrings
Other necessary changes were:
- indexing byestrings with a single index returns an integer on Python.
These indexing operations have been replaced with a one-element
slice: x[0] -> x[0:1] or x[-1] -> [-1:] so at to return a bytestring.
- raw_hash.encode('hex_codec') replaced with binascii.hexlify(raw_hash)
- str(integer) -> b'%d' % integer
- 'string_escape' codec replaced with 'unicode_escape' (which was
backported to python 2.7). Strings decoded with this codec were then
immediately re-encoded with UTF8.
- Calls to map() intended to execute their contents immediately were
unwrapped or converted to list comprehensions, since map() is an
iterator and does not execute until iterated over.
hg-fast-export.sh has been modified to not require Python 2. Instead, if
PYTHON has not been defined, it checks python2, python, then python3,
and uses the first one that exists and can import the mercurial module.
From PEP 394 [1]:
* python2 will refer to some version of Python 2.x.
* end users should be aware that python refers to python3 on at least
Arch Linux (that change is what prompted the creation of this PEP),
so python should be used in the shebang line only for scripts that
are source compatible with both Python 2 and 3.
So to make sure that we run correctly on a system where python refers
to python3 and avoid problems like issue #11 we change the shebangs.
[1] https://www.python.org/dev/peps/pep-0394/
os.popen() uses the shell, this is dangerous when the branch-name
contains characters which are interpreted by the shell, therefore switch
to subprocess.check_output() which doesn't involve the shell.
This closes issue #66.
When a mercurial repository does not use utf-8 for encoding author
strings and commit messages the "-e <encoding>" command line option
can be used to force fast-export to convert incoming meta data from
<encoding> to utf-8.
When "-e <encoding>" is given, we use Python's string
decoding/encoding API to convert meta data on the fly when processing
commits.
After an update to Mercurial 2.3 the module 'repo' was removed and the
program crashed when trying to convert a repository. I checked the
imports with 'pyflakes' and removed all unused ones, repo (among
others) was never used.
http://www.selenic.com/repo/hg/rev/1ac628cd7113#l9.1
Signed-off-by: Frej Drejhammar <frej.drejhammar@gmail.com>
Reported-by: Cole Robinson <crobinso@redhat.com>
Thank's to Cole Robinson for reporting the bug and providing a fix
which was adapted to this patch.
The original bug report:
I was recently converting a few mercurial repositories to git, and
noticed certain commits had their date reset to Jan 1 1970.
An example repo:
http://hg.fedorahosted.org/hg/virt-manager
An example commit:
http://hg.fedorahosted.org/hg/virt-manager/rev/41182500ddef
After some poking, it seems the culprit was that the "author:" was
surrounded by quotation marks
Add -M, --default-branch <branch_name> to allow user to set
the default branch where to pull into
Signed-off-by: Fabrizio Chiarello <ponch@autistici.org>
In git 1.6.0, most git tools with a dash in the name will no
longer be installed in $bindir. This patch makes hg-fast-export
use the "git <command>" form so it will work even if "git" is
the only piece of git machinery in the user's PATH.
On the other hand, the "git <command>" form does not help for
sourcing a shell script (with "."). So use the full path to
source "git-sh-setup".
Signed-off-by: Jonathan Nieder <jrnieder@uchicago.edu>
The get_branch() function's purpose is to detect whether a mercurial
branch name actually should be considered the default branch.
Sanitizing branch and tag names for git is done in sanitize_name().
Noted by Jonathan Nieder.
This reverts commit cdfdae36c8.
Since space doesn't conform to GIT branches name standards,
it should be replaced or with another character.
Signed-off-by: Felipe Zimmerle <felipe.zimmerle@indt.org.br>
Signed-off-by: Rocco Rutte <pdmef@gmx.net>
By default, the key is not changed. This will allow us for fixing up the
off-by-one issue with marks restored using load_cache().
Signed-off-by: Rocco Rutte <pdmef@gmx.net>
Unfortunately, I can't do 'import hg-fast-export' from python itself, so
we need to move some common methods into 'hg2git.py' which is to be used
as a library for common hg->git routines.
Signed-off-by: Rocco Rutte <pdmef@gmx.net>
It's pointless for many branches to print the validation message for the
first revision already; the same counts for incremental runs.
Signed-off-by: Rocco Rutte <pdmef@gmx.net>
Now we have three methods of feeding out changes
1) full for first revision or
2) thorough delta for merges (compare checksums with all parents) or
3) simple delta else (only got with manifest)
This requires some cleanup so that we have only place where we actually
call the appropriate dumping method.
The export_file_contents() method now also sorts its file list before
writing out anything as this seems to speed up hg data retrival a bit.
Signed-off-by: Rocco Rutte <pdmef@gmx.net>
For the mutt and hg repos, it didn't make a difference, but attempting
to run the conversion on the opensolaris repo looks like this is needed.
When we attempt to export some commit, special-case the revision number
0 and export all files the manifest has while labeling this a "full
revision export". Otherwise we do what we did before labeling this a
"delta revision export".
Signed-off-by: Rocco Rutte <pdmef@gmx.net>
Previously, when no head was present under .git/refs/heads, we simply
died as we couldn't open the file. Now, simply return None in case we
cannot read from it.
Signed-off-by: Rocco Rutte <pdmef@gmx.net>
The current regex may leave us with keys/values having trailing/leading
spaces in all flavours which will break lookup. Solution: strip() key
and value.
Signed-off-by: Rocco Rutte <pdmef@gmx.net>
By allowing spaces in keys we allow for (re-)mapping complete lines
like "Joe User <joe@host>" to be mapped to something else.
Signed-off-by: Rocco Rutte <pdmef@gmx.net>
This broke incremental imports as hg2git.sh wrapper overwrites headsfile
with current values after the import is done.
Signed-off-by: Rocco Rutte <pdmef@gmx.net>
IMHO it's highly unusual to have these lines in hg projects but who
knows. As it's slow to parse these types of lines (with regex), it's
disabled by default and the 'author' command of git-fast-import isn't
used at all.
It can be enabled by giving the -s switch to hg2git.sh.
Signed-off-by: Rocco Rutte <pdmef@gmx.net>
