Success!

As of today, I have officially made a working SWIG representation of a Bio++ object. (And it's about bloody time.)

I was having some linking errors, so I spent some time today rereading SWIG docs (specifically the bits about Python). Turns out that the errors I was having were mentioned explicitly in them. When I linked together the .so file, I needed to make sure that all its dependencies were there.

So here's the commands I used:

$ swig -python -c++ -I../../../../contrib/biopp/utils/Utils/ Exceptions.i
$ g++ -c -fpic Exceptions.cpp
$ g++ -c -fpic TextTools.cpp
$ gcc -c -fpic Exceptions_wrap.cxx -I/usr/include/python2.6/
$ g++ -shared Exceptions.o Exceptions_wrap.o TextTools.o -o _bpp_exceptions.so

Exceptions depends on TextTools, hence the inclusion of that object file.

I'm not certain that -fpic is necessary--some of the SWIG docs indicate that this is only for some specific platforms, but not Linux. It certainly doesn't hurt things, though.

I think things will be easier if I can manage to put everything in utils into a single object file.

Happy Dance!

It's working! At least, the tiny little C++ program I had written is working. Basically, I needed to include the proper local .h files in the .i file's headers, but only within the %{ and %} delimiters. That way SWIG doesn't choke on them, but they're still included (verbatim) in the created .cxx file.

Now we try this on the actual Bio++ files...

Still Not Working...

Still lots and lots of compiler errors when I try to compile the wrapper function--even though SWIG itself runs without any errors at all.  They look like scoping and syntax errors, but I'm not sure what's causing them.  I've been working with very a simple C++ file, with a single object, and no namespace issues.  Using that, I get the same errors as when I try to compile the Bio++ wrappers.

One thing I have discovered--this is language invariant.  It doesn't matter if I try to use Java, Python, etc.--they all get the same errors.

Reread Chapters 5 and 6 of the SWIG docs today, very carefully.  Found very little of any relevance. Finally posted a note to the SWIG mailing list. I feel like an idiot doing so, but this needs to get done.

Convention

Spent most of today at the open source convention, thanks to Ellen and Leslie at Google. Managed to spare a couple of hours to back up a bit (again) and start exploring how namespaces interact with SWIG.

%#&*@!

And things were going so smoothly.

The xxx_wrap.cxx files that SWIG makes do not compile. I get about two pages worth of bugs, some of which seem to be syntax errors. This makes me unhappy, to say the least. What kind of tool outputs code that can't compile?

Back up a bit...I have to assume that I did something wrong in the .i file. The errors seem to indicate that this might be a namespace issue. (And unfortunately, namespaces are new enough that they weren't teaching them when I learned C++.) It's telling me that classes I try to compile aren't part of the bpp namespace. Looking through the original source, it seems that might be correct. All the .cpp files indicate that we're using the bpp namespace, so that we can access functions in it without having to precede everything with bpp::. But nothing actually seems to be in it. On the other hand, the function prototypes are in bpp in the .hpp files.

Honestly, I'm ready to just try to rip out all references to the namespace from the original source, and see if it works. I'm very unhappy with this--I'd hoped that I was all ready to roll, and to get some serious work done. Instead, I'm thinking that this is going to take me another week to get through. I'm not fulfilling what I've promised to do, and that infuriates me.

Operator Issue

Just solved something that's been plaguing me for a week. I figured out why the example code in the SWIG docs weren't working with operator overloading--namespaces. The docs showed %rename and %extend being used with little difficulty, but all their examples were small enough that they didn't use namespaces. And Xin's notes said to use %rename toward the beginning of the file. No effect...but as soon as I tried putting these statements inside the bpp namespace, it all worked. (And it only took me several hours to figure out...)

Only one issue is left--the "incomplete" string class. I've sent another note to the SWIG mailing list, this time without the problem being buried way down in the note. Hopefully someone will see it.

Going to take a dinner break right now, then come back to this later.

Still SWIGing

Here are the warnings that are left:

ApplicationTools.i:21: Warning(454): Setting a pointer/reference variable may leak memory.
ApplicationTools.i:22: Warning(454): Setting a pointer/reference variable may leak memory.
ApplicationTools.i:23: Warning(454): Setting a pointer/reference variable may leak memory.

BppString.i:13: Warning(402): Base class 'string' is incomplete.
/usr/share/swig1.3/typemaps/std_string.swg:17: Warning(402): Only forward declaration 'string' was found.

Number.i:19: Warning(362): operator= ignored

RGBColor.i:43: Warning(389): operator[] ignored (consider using %extend)
RGBColor.i:44: Warning(389): operator[] ignored (consider using %extend)

I'm inclined to ignore the possible memory leak in ApplicationTools. I still need to figure out the operator issue in RGBColor and Number. And then the BppString issue--I have no idea. My thought is that it's an issue with the SWIG library on which everything relies.

Have some information from Xin about the operator issue, and I have a couple of things to try. (And in any case, I'm not really sure I need to do anything here--if someone can't access Utils from Python or Perl, I don't think it's much of a loss.) That leaves the base 'string' class problem. Have posted to the SWIG mailing list, but have gotten nothing from them. (That might be because I did it as an addendum to an earlier question of mine, that ended up being a dumb question.)

Sunday Work

Did work today, because I missed some of the last work week due to being sick.

Now have .i files for everything in utils. Most of them compile. The ones that don't seem to have some pretty uniform problems, that I need to look into. For example, things that refer to Exceptions.h have problems. This is because Exceptions.h is a local file that refers to the standard C++ exception object. So I get Exceptions.i to compile no problem (by telling it to include the C++ version it inherits), but other .i files don't know where it is. I could tell them explicitly, but I get the feeling that there's some better way to do it. (Perhaps by using the %import keyword rather than %include?)

So yeah, was happy to do some *real* coding, instead of just mucking about with make files.

Interface Files

Lots more reading today, of SWIG docs.

The main concrete thing I did was to succeed in making a SWIG .i file from scratch, and then I used it to get to where I was yesterday--getting SWIG to work (albeit with warnings). And then I managed to figure out how to make it pay attention to dependencies and inheritance. This was a little tricky for the STL, but I found how to do it (there are special .i files that ship with SWIG, that you have to include).

However, now I'm getting this pair of error for BppString.i:

BppString.i:13: Warning(402): Base class 'string' is incomplete.
/usr/share/swig1.3/typemaps/std_string.swg:17: Warning(402): Only forward declaration 'string' was found.

And there seems to be very little online with regard to this. Finally ended up shooting off a question to the SWIG mailing list, hoping one of them would have an answer.

Regardless, I think I know how to make .i files now. (Part of that is from reading through Xin's code.) And that represents actual coding (of a sort), so I feel I can actually move forward.

Pjotr also mentioned that Xin dealt with the operator warnings messages, so hopefully he's already found an answer of some kind.

Baby Steps

So Pjotr recommended that I back up, and try to use SWIG on tiny bits of the code. At the same time, I realized that having a 35K-line .i file was insane, when "based" off a 300-line .h file (especially when about half of those 300 lines are comments). Xin has been making his own .i files, so I decided that I'd have a go at that.

And then, looking through SWIG docs, I find that that's not even necessary for some very simple .h files. There's a -module option that lets you go directly off of the .h file. So...can't hurt to try it. Here are the results, by header file (in utils). In the cases where it compiled sans error, I just left it blank:

AttributesTools
ApplicationTools
 ApplicationTools.h:96: Warning(454): Setting a pointer/reference variable may leak memory.
 ApplicationTools.h:100: Warning(454): Setting a pointer/reference variable may leak memory.
 ApplicationTools.h:104: Warning(454): Setting a pointer/reference variable may leak memory.
BppString
 BppString.h:57: Warning(401): Nothing known about base class 'string'. Ignored.
 BppString.h:57: Warning(401): Nothing known about base class 'Clonable'. Ignored.
BppVector
Clonable
ColorManager
 ColorManager.h:109: Warning(401): Nothing known about base class 'ColorManager<>'. Ignored.
 ColorManager.h:109: Warning(401): Maybe you forgot to instantiate 'ColorManager<>' using %template.
ColorSet
ColorTools
DefaultColorSet
 DefaultColorSet.h:54: Warning(401): Nothing known about base class 'AbstractColorSet'. Ignored.
DvipsColorSet
 DvipsColorSet.h:55: Warning(401): Nothing known about base class 'AbstractColorSet'. Ignored.
Exceptions
 Exceptions.h:59: Warning(401): Nothing known about base class 'exception'. Ignored.
FileTools
Font
 Font.h:55: Warning(401): Nothing known about base class 'Clonable'. Ignored.
FontManager
 FontManager.h:107: Warning(401): Nothing known about base class 'FontManager<>'. Ignored.
 FontManager.h:107: Warning(401): Maybe you forgot to instantiate 'FontManager<>' using %template.
 FontManager.h:164: Warning(401): Nothing known about base class 'FontManager<>'. Ignored.
 FontManager.h:164: Warning(401): Maybe you forgot to instantiate 'FontManager<>' using %template.
GraphicDevice
IOFormat
KeyvalTools
 KeyvalTools.h:56: Warning(401): Nothing known about base class 'Exception'. Ignored.
MapTools
MolscriptColorSet
 MolscriptColorSet.h:53: Warning(401): Nothing known about base class 'AbstractColorSet'. Ignored.
Number
 Number.h:75: Warning(362): operator= ignored
PGFGraphicDevice
 PGFGraphicDevice.h:58: Warning(401): Nothing known about base class 'GraphicDevice'. Ignored.
RColorSet
 RColorSet.h:53: Warning(401): Nothing known about base class 'AbstractColorSet'. Ignored.
RGBColor
 RGBColor.h:117: Warning(389): operator[] ignored (consider using %extend)
 RGBColor.h:128: Warning(389): operator[] ignored (consider using %extend)
 RGBColor.h:60: Warning(401): Nothing known about base class 'Clonable'. Ignored.
StringTokenizer
SVGGraphicDevice
 SVGGraphicDevice.h:58: Warning(401): Nothing known about base class 'GraphicDevice'. Ignored.
TextTools
XFigGraphicDevice
 XFigGraphicDevice.h:60: Warning(401): Nothing known about base class 'GraphicDevice'. Ignored.

Everything compiled--this were all warnings. Many of them are just inheritance problems. Some are for other classes in utils (e.g. GraphicDevice, Clonable), and others are for standard C++ classes (e.g. string, exception). This should be easy to fix.

What I'm more worried about is the template stuff. Also strange are errors about the [] and = operators. For the former, it talks about using the %template directive in SWIG. That's a good place to start looking.

Finally, I feel like I'm moving forward again.

Backing Up...

Yesterday was a sick day. Got food poisoning from a party over the weekend. Not fun.

Today spent a lot of time trying to figure out why SWIG only gives me a single error line, without any substantial results. Also got reminded by Hilmar to e-mail weekly updates to everyone, and so did so.  (Thought the blog sufficed, but nope.) But then David offered to help, and now we have a discussion going. This might prove to be my lifeline.

Tomorrow's project is to start some smaller projects with SWIG. I've managed to SWIGify some very basic code, but I'm going to try a bit bigger now, and see what I get.

Git Working

I think I have Git under control. I have my own branch (biopp) inside of my own fork, and I'm successfully compiling things with very little effort. Pushing and pulling all seems to work, as do Pjotr's CMake macros.

The downside--tried using SWIG on more .i files: this time from NumCalc. (Aside: it occurred to me that I don't necessarily need to actually SWIGize Utils, since no one calls it directly, anyway.) The result was exactly the same. Something's not working, somewhere, and I still don't know why. I've sent queries to both the CMake and SWIG mailing lists, but so far no one's replied. Gotta keep plugging, and hope the answer presents itself soon.

God Dammit

I was using Git wrong. This is why many of Pjotr's messages have been confusing to me, and no doubt many of mine have been for him. This is all supposed to be done in a branch of biolib, not in its own tree altogether.

Of course, now that I've done things right, I can't figure out how to push back up. Somewhere, Git is just crashing out. Looking for help online seemed to indicate that there's a mismatch between the version I pulled, and what I'm trying to push. But nope--repulling doesn't help.

Meanwhile, I did manage to recreate my .i files. And guess what? SWIG still claims there's a syntax error, right on the same line.

Bleah. New day tomorrow.

Enter SWIG

Successfully used SWIG on an example file lifted from the SWIG documentation. It took quite an effort to get it to work, and I'm not really sure what I did to make it function. I just deleted everything and tried again, and maybe I got the order of commands correct. Don't know, and that disturbs me. I don't like inconsistent programs. (Or, more accurately, programs that are perfectly consistent but whose patterns I can't recognize.)

Also managed to use CMake on utils in order to make a bunch of .i files for SWIG. However I was unable to get SWIG to treat them properly. It kept complaining of a syntax error. The line number varied depending on the exact .i file, but the offending line is always:

namespace std __attribute__ ((__visibility__ ("default"))) {

I'm afraid that I'm going to need to dig deep into the SWIG docs to figure out what the hell this is. Not what I wanted, but oh well.

Decent Amount

Actually got a fair amount done today.

1. Used CMake to make an installation shared library, in my /usr/local/lib/ directory. Incorporated that into the test code I got working yesterday, and it worked perfectly. (I was actually kind of surprised that it did.)
2. Used code from the CMake FAQ (http://www.cmake.org/Wiki/CMake_FAQ) to modify CMakeLists.txt to do SWIG mappings. Took some effort, and it didn't work. While CMake ran fine, make complained about missing targets. The code they have up there doesn't have much in the way of comments to help me figure out what went wrong.
3. Learned another trick in Git: downloading submodules. Pjotr has written some packages for biolib to help with the SWIG mapping. With luck, that'll help me do what I need to. Tomorrow I dive fully into trying to understand that.

Yesterday's Progress

Was out last night, and didn't get back until late. Hence, the post-facto update now.

Actually managed to use Bio++ for the first time, yesterday. Got some code up and running that calls the library to make a calculation. This was harder than it should have been, because of dependency hell issues.

Also, managed to push my changes to the project (i.e. the addition of CMakeLists.txt) up to github. So I'm making slow progress there, too.

Things still going more slowly than I want them too, though. Hoping that this is just a rough patch that will be over soon.