Autor Thema: Merging oc.se -> oc.de  (Gelesen 10291 mal)

Offline OlofL

  • Nano
  • **
  • Beiträge: 70
  • OC SE Team
Merging oc.se -> oc.de
« am: 28. Dezember 2010, 22:57:26 »
A subproject of the task to merge features of the two main source versions have started. The pilot for this will be oc.se. The outline of this subproject is to:
- Create a branch (se-dev) from the oc.de source (oc-server). From a version control perspective this can be seen as a private branch for isolated development. The word "private" only means that the oc.de (oc-server) is unaffected during the development.
- Add features from the current oc.se source into the new se-dev-branch
- During this time work can still be done on the oc.de source (oc-server) and this can be merged to the new se-dev-branch.
- When the new se-dev-branch is stable enough for production oc.se will go live on this
- After the go live, this branch will be reintegrated back in the oc.de source (oc-server) and both oc.de and oc.se will go live on a common source.

The expected outcome of this subproject:
- Since oc.se is a branch off the oc.pl source, we will have a much better understanding of the challenges that lay ahead of us after this, for example a database upgrade script.
- We will have one branch less to think about, the two swedish developers can work on the joint de-se code after this
- The de source will be fully translated to at least swedish and probably some more languages.

There will be decision points throughout this subproject whether if it wise to continue or not.

If you would like to join this subproject, you are welcome.

Offline OlofL

  • Nano
  • **
  • Beiträge: 70
  • OC SE Team
Re: Merging oc.se -> oc.de
« Antwort #1 am: 29. Dezember 2010, 13:38:15 »
I've already gotten a few questions, one of them is about switching to a distributed version control system like git or Mercurial.

Regarding the version control I agree that a distributed system like Mercurial  or git is much more fit for the kind of work we are doing. But the target source is the oc.de master and it is that version handling system that would benefit from being updated to git och Mercurial. However, I judge it as unlikely that we will succeed in changing both the version control and lifting features at the
same time. Lifting features from oc.se -> oc.de is a very concrete thing to do with limited risk and less dependencies on people that might go in and out of the project.

Offline totsubo

  • Nano
  • **
  • Beiträge: 17
Re: Merging oc.se -> oc.de
« Antwort #2 am: 30. Dezember 2010, 15:00:44 »
I've gone and created a Google Project for us. This gives us access to an issue tracker, Wiki, and more. It's located here:

http://code.google.com/p/opencaching/issues/list

The only drawbacks AFAICT is that we'll all need a Google account to create issues, etc. FYI anyone can create issues but only project members can update bug statuses or create Wiki pages.

I'm also looking into installing Bugzilla on my opencaching node. I'm thinking that it's probably not necessary and the Google Project will meet all our needs but I'll keep working on installing Bugzilla just in case.

To be fair I have absolutely no experience using Google Projects but it seems to offer us two things we want, an issue tracker and a Wiki, for free. If anyone has any comments, pro or con, about Google Projects I'm happy to hear them.

For those that want to be added to the list of project members please raise an issue using the issue tracker and I will go ahead and add you as a project member.

Offline totsubo

  • Nano
  • **
  • Beiträge: 17
Using a Distributed revision control system
« Antwort #3 am: 30. Dezember 2010, 15:15:52 »
I've already gotten a few questions, one of them is about switching to a distributed version control system like git or Mercurial.

[snip]

However, I judge it as unlikely that we will succeed in changing both the version control and lifting features at the
same time.

Can I recommend the following:

- Create an SVN se-dev branch as you suggest
- Make the branch read-only
- Create a git repository on github (or other git repository site) with a copy of the se-dev branch
- Project members then do development using git and the github branch
- When the project is complete:
  - make SVN se-dev branch writable by project manager
  - project manager pushes all changes back into the SVN se-dev branch
  - make SVN se-dev branch read-only again
  - work starts on merging se-dev back into oc.de main branch. (If little changes on the oc.de main branch this should be easy)

I use git for personal projects and have only a little experience with it but I think the method above would work. I have nothing against using mercurial; I simply prefer git since I am more familiar with it.

Offline OlofL

  • Nano
  • **
  • Beiträge: 70
  • OC SE Team
Re: Merging oc.se -> oc.de
« Antwort #4 am: 01. Januar 2011, 20:23:32 »
The branch has been created so work is now started.

Offline totsubo

  • Nano
  • **
  • Beiträge: 17
Re: Merging oc.se -> oc.de
« Antwort #5 am: 05. Januar 2011, 15:38:10 »
I've set up the git repository. It's a copy of the oc-se svn branch that Oliver set up.

git@github.com:totsubo/se2de-merge.git

I just took the snapshot so it should be up-to-date. The link is read/write; we're probably all new to git so I'm sure we'll all make a few mistakes writing to the repository :)

We should now stop using the SVN repo.

Offline OlofL

  • Nano
  • **
  • Beiträge: 70
  • OC SE Team
Re: Merging oc.se -> oc.de
« Antwort #6 am: 06. Januar 2011, 00:45:18 »
The plan is to regards the oc-se-branch in the svn-repo as temporary read only and to do the work on the branch in git. At a later time we can push things back from git to this branch and then merge from the branch into the svn-trunk.

oliver

  • Gast
Re: Merging oc.se -> oc.de
« Antwort #7 am: 07. Januar 2011, 21:42:52 »
Hi totsubo,

in our teamspeak meeting, we talked about the option to switch to git now or in a few months and we decided to start with the existing SVN repository. And invest our work in implementation and not "administration" for the moment, since there are no drawbacks with SVN at the moment. When the new code is used by more than one node, git has the advantage of better code customisation possibilities (if i understand you right), but at the moment we should focus in writing code that is commonly usable. The (oc.de) development systems and (oc.de) production enviroments (update process) are configured to use SVN and switching to GIT requires some preperation. Since i have little experience with git, i need to learn some things about the system and find a new client for linux and windows.

If you only migrate the se2de-branch from SVN to GIT, we loss the advantage to be able to use the existing translation system with both codes and only one "translation repository". So, switching instant to git is no real option in my opinion and we should delay the git-migration to that point were the se-branch is merged back to the trunk.

However, if git migration has benefits at the moment, we should move as soon as possible and i will change our development and production systems to use git. But then, the entire SVN repository should be moved.

@Olof: Is my summary correct? Should we switch now to git?

@totsubo:
- Is the git-repository public read-writeable? Until now, only well known users have access to our repositories. This simplifies license management since we do not redistribute the code "to everyone".
- Is there a bugtracker with "file-structure integration" available at github?
- Is there a email notification for changes available?
- How is the repository backed up?


Regards,
Oliver

Offline totsubo

  • Nano
  • **
  • Beiträge: 17
Re: Merging oc.se -> oc.de
« Antwort #8 am: 08. Januar 2011, 03:23:38 »
Hi Oliver and thanks for the update on the teamspeak. I'll try and answer you questions as best as possible. If I miss anything please let me know.


in our teamspeak meeting, we talked about the option to switch to git now or in a few months and we decided to start with the existing SVN repository. And invest our work in implementation and not "administration" for the moment, since there are no drawbacks with SVN at the moment. [...] The (oc.de) development systems and (oc.de) production enviroments (update process) are configured to use SVN and switching to GIT requires some preperation. Since i have little experience with git, i need to learn some things about the system and find a new client for linux and windows.

For the oc.de work I agree that for now it probably makes more sense for you to keep going with SVN, especially if you have some background admin tool or upgrade tools configured to use SVN. The priority for oc.de should be the code merge. Switching to git for you will incur some some initial investment in time/etc. and that will take away from the time you have for the actual code work.


If you only migrate the se2de-branch from SVN to GIT, we loss the advantage to be able to use the existing translation system with both codes and only one "translation repository". So, switching instant to git is no real option in my opinion and we should delay the git-migration to that point were the se-branch is merged back to the trunk.


First, I haven't looked at the oc.de code so I'm just guessing, but isn't the translation done like on oc.pl? There is one big translation file for each language, no?

Just to make sure I understand, you are worried that if we put the se2de-branch in git you will no longer have access to the code changes, is that correct?

You don't need to worry about this since our end gola is to put all our work back into the SVN se2de-branch. We will do all our merging using git *but* once we are done we will commit everything back into SVN and you will have access to it as you do now.


@totsubo:
- Is the git-repository public read-writeable? Until now, only well known users have access to our repositories. This simplifies license management since we do not redistribute the code "to everyone".
- Is there a bugtracker with "file-structure integration" available at github?
- Is there a email notification for changes available?
- How is the repository backed up?


1- It is world readable but we control the write access.

2- There is an issue tracker at github but we are not using it. Instead we are using the Google Project issue tracker here: http://code.google.com/p/opencaching/issues/list . The github issue tracker is very primitive and the Google one much better.

2.1 - I'm not sure what 'file-structure integration' means for an issue tracker. If you explain it a bit more I can try to answer the question :)

3- The issue tracker has email notification. github has some form of email notification on commits but I haven't played with it.

4- Ah ... this is one of the great things about git.
  4.1 - github probably does regular backups. I'm pretty sure they are solid. You can trust them as much as you trust your sever hosting.
  4.2 - But even better, using a distributed system like git *everyone* has a copy of the repository on their local hard drive! If github goes down we don't really lose anything since we can rebuild the full repository (and all branches) from anyone. (sorry if my explanation is probably not very good)

I hope that explains things. If not I  am sure Olof can probably explain things better than I can :)

oliver

  • Gast
Re: Merging oc.se -> oc.de
« Antwort #9 am: 08. Januar 2011, 09:36:32 »

If you only migrate the se2de-branch from SVN to GIT, we loss the advantage to be able to use the existing translation system with both codes and only one "translation repository". So, switching instant to git is no real option in my opinion and we should delay the git-migration to that point were the se-branch is merged back to the trunk.


First, I haven't looked at the oc.de code so I'm just guessing, but isn't the translation done like on oc.pl? There is one big translation file for each language, no?

Just to make sure I understand, you are worried that if we put the se2de-branch in git you will no longer have access to the code changes, is that correct?

Strings to translate are scanned from database lookup tables and from each php file. The translations are stored in the database and later cached in gettext()-files. There is a screenshot-posting in the oc.pl-forum. Please have a look at it (my notebook has trouble to login there at the moment, sorry).

The issue is that the scan-engine needs to have access to all codes or you will end up in 2 translation repositories which you need to merge again.

You don't need to worry about this since our end gola is to put all our work back into the SVN se2de-branch. We will do all our merging using git *but* once we are done we will commit everything back into SVN and you will have access to it as you do now.

But then, you must wait with translations until that merge-back is done or we need to maintain 2 translation repositories. As of my experience, you should begin translation to one foreign language as soon as possible within the development cycle to ensure that everything is marked to be translated. And having 2 translation repositories is would be very bad ...

... so the options is "switch all to git instant" or wait for the git-migration.

What i have not known is that Olof uses git for his development on the top of our SVN repository and that he wants to wait for merge-back until he has finished the work. So git seems to be an advantage for his development. If thats the case, we should switch to git now with all the code.

@totsubo: Can we move the entire svn repository to git with the same structure as it is? Including trunk, branches, tags? Or are there any changes in the structure required? What i want to do in near future is to use the dev-branch before commiting to the trunk. And Olof suggested to use svn-externals for 3rd-party components. This means we have another top level directory "3rd-party" and inside this we store 3rd-party components e.g. 3rd-party/smarty.

- Is the git-repository public read-writeable? Until now, only well known users have access to our repositories. This simplifies license management since we do not redistribute the code "to everyone".

1- It is world readable but we control the write access.

Have you planned to keep it public readable? I dont know how the other developers think about that point - for me both solutions are okay (but i prefer a protected repository, slightly). Because this is a big question, we should continue the licensing discussion from oc.pl forum in new topic here (and of course, we should find a decision in that point). The suggestions was AGPL licsense, the concern was the question if all used components fit that license. Maybe it makes sense to introduce the svn-externals first, since license control is easier in that case. So we can keep all our code in trunk and branches and codes with other licenses in "3rd-party".

- Is there a bugtracker with "file-structure integration" available at github?

2- There is an issue tracker at github but we are not using it. Instead we are using the Google Project issue tracker here: http://code.google.com/p/opencaching/issues/list . The github issue tracker is very primitive and the Google one much better.

I didnt work with these tools until now. I think it would be very comfortable to be able to attach an issue to one specific file (and the tracker should be able to list the files automatically). When working on one file, the developer could query "what else to do here" and there is another simple task on that file, he could do it ...

... and of course, i think its not good to spread the development tools over too much different plattforms, since we get some problems in administration of the user accounts and a developer needs to manage many accounts and websites ... until now, there are 3 different websites (forum, git, tracker). Have i missed one?

- Is there a email notification for changes available?

3- The issue tracker has email notification. github has some form of email notification on commits but I haven't played with it.

Sounds good :)

- How is the repository backed up?

4- Ah ... this is one of the great things about git.
  4.1 - github probably does regular backups. I'm pretty sure they are solid. You can trust them as much as you trust your sever hosting.
  4.2 - But even better, using a distributed system like git *everyone* has a copy of the repository on their local hard drive! If github goes down we don't really lose anything since we can rebuild the full repository (and all branches) from anyone. (sorry if my explanation is probably not very good)

4.1 => i dont trust myself regarding the backup ;)
4.2 => so, if the repository has 2 GB with all history, everyone requries to load that 2 GB?

What i missed in SVN is the opportunity to delete a file including its history ... there are some big files that mess up the repository with the history, but the history of that files is not requried.

I hope that explains things. If not I  am sure Olof can probably explain things better than I can :)

I think we all have some stress to pharse the complex things in english ... its challanging to explain it in the native language, write it down and translate it makes it even more difficult ... but i think i understand your postings ...
« Letzte Änderung: 08. Januar 2011, 09:44:58 von oliver »

Offline mic@

  • Vereinsmitglied
  • Large
  • *
  • Beiträge: 6440
  • oc-only Verstecker
Re: Merging oc.se -> oc.de
« Antwort #10 am: 08. Januar 2011, 10:39:00 »
Zitat von: oliver
There is a screenshot-posting in the oc.pl-forum. Please have a look at it (my notebook has trouble to login there at the moment, sorry).
My notebook is working  8)
==> http://forum.opencaching.pl/viewtopic.php?f=45&t=5579

Offline OlofL

  • Nano
  • **
  • Beiträge: 70
  • OC SE Team
Re: Merging oc.se -> oc.de
« Antwort #11 am: 08. Januar 2011, 10:58:39 »
git is really not an issue outside the subproject. See upon git as a tool used for lifting issues only. The german master should *not* be moved to git now, it would complicate things a lot and probably destroy the work we have already begun.

Zitat
... so the options is "switch all to git instant" or wait for the git-migration.
The answer to that is "wait for the git-migration". What we are doing now is something else, we are only using git as a tool during work. When things are more stabilized in this branch, the changes will be checked into svn and the translation tool can be used.

Apart from using it as a tool, we also gain some experince that will come in handy in the future if we decide to switch version handling system to a distributed system like git or Mercurial.

Offline OlofL

  • Nano
  • **
  • Beiträge: 70
  • OC SE Team
Re: Merging oc.se -> oc.de
« Antwort #12 am: 08. Januar 2011, 20:46:21 »
The updating scripts for the database are finished in their first version. They are sufficient to convert a developer pl-branch database to a de-branch version. However there are some remaining issues related to neccessary development work needed. There are some information on the user, some on caches and some on cache logs that are lost until these issues are solved.

We should treat these scripts as working documents up until the moment we perform the update. They must be run in order and they must be run as a user with priviledges enough to drop triggers. Triggers should be installed prior to running them, but needs to be reinstalled again afterwards.

Get in touch if you want to have a look at the scripts.

Offline totsubo

  • Nano
  • **
  • Beiträge: 17
Re: Merging oc.se -> oc.de
« Antwort #13 am: 09. Januar 2011, 14:46:27 »
Oliver,

We will be merging in some new functionality but of course we want to make it configurable since not eve node will want the new functionality.

Currently, what is the mechanism in the oc.de code to control functionality is enabled/disabled on an oc node? We want to use the same method.

Thanks

oliver

  • Gast
Re: Merging oc.se -> oc.de
« Antwort #14 am: 09. Januar 2011, 15:12:26 »
Currently, what is the mechanism in the oc.de code to control functionality is enabled/disabled on an oc node? We want to use the same method.

configuration is done in config2/settings.inc.php
defaults and explanation should be palced in config2/settings-dist.inc.php

At runtime, settings-dist.inc.php is included first and after that settings.inc.php is included. This way you can customize all values that were preset in settings-dist.inc.php

Of course, the number of configuration params increased a lot in the last months. And i am not sure if there are some keys that should be restructured. Depending on the feature you implement, you can choose a key in $opt ...

e.g.
$opt['modules']['review']['enabled'] = true;
$opt['modules']['reveiw']['requiredfinds'] = 5;

Regarding the configuration system we have the following todos in the future:
1) Check if the structure of $opt is consistent and move some configuration keys to a better subkey (restructure)
2) Use the config2-configuration for the old template system, too (there all is still configured in lib/settings.inc.php, without settings-dist)
3) Maybe reducing the config file to the absolutely required params and placing the rest within the database would be good (and use some file based caching)