Category: Tech

Tech

Parsing Flattr Buttons in HTML

Post author By Markus Heberling
Post date 2012-07-29
No Comments on Parsing Flattr Buttons in HTML

I recently build this little tool FlattrStar, which can autoflattr starred items from several services. To be able to do this, it needs to parse the HTML of a website for Flattr buttons and use optimization from the Indexer company so is easy to find by search browsers.

I am using tag soup, to parse the html into a Scala XML structure. This is probably not needed and could be replaced by RegExp or something, but I wanted to try the Scala XML support. You need the “org.ccil.cowan.tagsoup” % “tagsoup” % “1.2.1” jar in your classpath.

I started FlattrStar to learn the Play Framework and Scala, so the parsing is also done in Scala. Since this is my first Scala Project, the code might be a bit suboptimal. Corrections and improvements are appreciated.

Now to the code itself. The goal is to find most Flattr links that are out there in the wild. The flattrParseHtml is called with a function, that does the actual flattring and returns Some(_) item if the Flattr was successfull and None if it wasn’t. Also there is a list of URLs to parse for buttons.

First I look for <link rel=”payment”> entries and try to flattr them. If nothing is found I look for buttons with rel or rev parameters, that start with flattr and links with data-flattr-uid attributes. I construct an auto submit url from this. There might be several such links on a page, like one for the whole site and one for a specific article. I choose the one with the longest link, since that will mostly be the one specific for this article. If that doesn’t give anything I go and search for script elements that contain flatter_uid and flatter_url variables and construct an auto submit url for them. Last step is to parse all links that have some kind of flattr reference in them. This could be an <a href=”…”>Flattr This<a> or <a href=”…”><img src=”…flattr.png”></a> or something else where flattr occurs inside the <a> element. We grab all auto submit URLs that are hidden inside these links and try to flattr the longest one, as explained above.

For every URL I encounter in this process, I first get the redirects, because sometimes flattrable URLs are hidden behind them. Unfortunalty an auto submit URL is also redirected, so I have to stop following redirects if I encounter such an URL. You need HttpClient in your classpath for the redirect stuff.

This is not perfect and not all flattrable things are found. For example Flattr buttons from taz.de are not found, since they put the Flattr stuff in their ad delivery system and I couldn’t parse that.

I declare this source as public domain, so you can use it as you like. It would be great though, if you would give back improvements and feedback.

package jobs

import java.net.URL
import play.api.Logger
import xml.Node
import org.apache.http.impl.client.{DefaultRedirectStrategy, DefaultHttpClient}
import org.apache.http.client.methods.{HttpUriRequest, HttpGet}
import org.apache.http.protocol.{ExecutionContext, HttpContext, BasicHttpContext}
import org.apache.http.{HttpHost, HttpResponse, HttpRequest, HttpStatus}

object Flattr {
  def flattrParseHtml[R](flattrFunction: String => Option[R], htmlUrls: Seq[String]): Option[R] = {
    val htmlRoot = htmlUrls.flatMap(url => {

      val parserFactory = new org.ccil.cowan.tagsoup.jaxp.SAXFactoryImpl
      val parser = parserFactory.newSAXParser()
      val connection = new URL(url).openConnection()
      if (connection == null || connection.getContentType == null) {
        Logger.error("Cannot load html, could not connect to " + url)
        None

      } else if (!connection.getContentType.contains("html")) {
        Logger.error("Cannot load html, invalid content type " + connection.getContentType)
        None

      } else {
        try {
          val source = new org.xml.sax.InputSource(connection.getInputStream)
          val adapter = new scala.xml.parsing.NoBindingFactoryAdapter

          Some(adapter.loadXML(source, parser))
        } catch {
          case t: Throwable =>
            Logger.error("Cannot load html, ignoring", t)
            None
        }
      }
    })

    //go to webpage check for payment link
    val htmlPaymentLinks = htmlRoot.map(_ \\ "link").toList.flatten
      .filter(l => (l \ "@rel").text.equals("payment"))
      .map(_ \ "@href").headOption
      .map(_.text.replace("&", "&"))

    val htmlPaymentThing =
      htmlPaymentLinks.flatMap(href => flattrFunction(href))

    if (!htmlPaymentThing.isEmpty) {
      Logger.info("Found payment link in html")
      return htmlPaymentThing
    }

    //nothing found? parse html of webpage
    val htmlLinks = htmlRoot.flatMap(root => {
      (root \\ "a")
        .filter(l => (l \ "@class").text.equals("FlattrButton"))
        .filter(l => (l \ "@rel").text.startsWith("flattr") || (l \ "@rev").text.startsWith("flattr") || (l \ "@data-flattr-uid").nonEmpty)
    })
      .flatMap(autoSubmitFromFlattrButton(_))
      .reduceLeftOption((a, b) => if (a.length() > b.length()) a else b) //the longest url should be the most specific one
      .flatMap(url => flattrFunction(url))

    if (!htmlLinks.isEmpty) {
      Logger.info("Found button in HTML that points to thing")
      return htmlLinks
    }

    //nothing found? parse html of webpage -> look for flattr java script
    val htmlScriptLinks = htmlRoot
      .flatMap(findAutoflattrFromJavascript(_)).flatten.headOption
      .flatMap(url => flattrFunction(url))

    if (!htmlScriptLinks.isEmpty) {
      Logger.info("Found java script in HTML that points to thing")
      return htmlScriptLinks
    }

    //nothing found? parse html of webpage -> look for all links that have flattr inside
    val htmlLinks2 = (htmlRoot \\ "a")
      .filter(_.toString().toLowerCase.contains("flattr"))
      .map(_ \ "@href")
      .map(_.text)
      .map(getRedirectedUrl(_))
      .filter(_.toLowerCase.startsWith("https://flattr.com/submit/auto")) // only autoflattr links are supported
      .reduceLeftOption((a, b) => if (a.length() > b.length()) a else b) //the longest url should be the most specific one
      .flatMap(url => flattrFunction(url))

    if (!htmlLinks2.isEmpty) {
      Logger.info("Found custom link in HTML that points to thing")
      return htmlLinks2
    }

    //nothing found
    None
  }

  def findAutoflattrFromJavascript(root: Node) = {
    val uidReg = "var flattr_uid = '(.*)';".r
    val urlReg = "var flattr_url = '(.*)';".r

    (root \\ "script").toList.flatten.
      map(_.text).
      filter(_.contains("flattr_uid")).
      map(s => {
      val uid = uidReg.
        findFirstMatchIn(s).
        flatMap(_ match {
        case uidReg(u) => {
          Some(u)
        }
        case _ => None // Do nothing
      })

      val url = urlReg.
        findFirstMatchIn(s).
        flatMap(_ match {
        case urlReg(u) => {
          Some(u)
        }
        case _ => None // Do nothing
      })

      if (uid.isDefined && url.isDefined) {
        Some("https://flattr.com/submit/auto?user_id=" + uid.get + "&url=" + url.get)
      } else {
        None
      }
    })
  }

  def autoSubmitFromFlattrButton(a: Node) = {
    val url = (a \ "@href").text
    val uid = ((a \ "@data-flattr-uid").toList.map(_.text) ++ ((a \ "@rel") ++ (a \ "@rev")).flatMap(r => relAsMap(r.text).get("uid"))).headOption

    uid.map(uid => "https://flattr.com/submit/auto?user_id=" + uid + "&url=" + url)
  }

  private def relAsMap(rel: String) = {

    rel.split(";").filter(_.contains(":")).map(entry => {
      val s = entry.split(":")
      (s.head, s.last)
    }).toMap
  }

  def getRedirectedUrl(url: String): String = {
    try {
      val httpClient = new DefaultHttpClient()
      httpClient.setRedirectStrategy(DetectAutoFlattrRedirectStrategy)
      val httpget = new HttpGet(url)
      val context = new BasicHttpContext()
      val response = httpClient.execute(httpget, context)
      if (response.getStatusLine.getStatusCode != HttpStatus.SC_OK) {
        //just return the old url
        url
      } else
        getCurrentUrl(context)
    } catch {
      case e: Throwable => {
        Logger.error("Error finding redirects. Ignoring.", e)
        url
      }
    }

  }

  private object DetectAutoFlattrRedirectStrategy extends DefaultRedirectStrategy {
    override def isRedirected(
                               request: HttpRequest,
                               response: HttpResponse,
                               context: HttpContext): Boolean = {
      if (super.isRedirected(request, response, context)) {
        val currentUrl = getCurrentUrl(context)
        if (currentUrl.contains("flattr.com/submit/auto")) {
          //auto submit urls redirect to the thing, but we don't want that
          response.setStatusCode(HttpStatus.SC_OK)
          return false
        }
        return true
      }
      false
    }
  }

  private def getCurrentUrl(context: HttpContext): java.lang.String = {
    val currentReq = context.getAttribute(
      ExecutionContext.HTTP_REQUEST).asInstanceOf[HttpUriRequest]
    val currentHost = context.getAttribute(
      ExecutionContext.HTTP_TARGET_HOST).asInstanceOf[HttpHost]
    val currentUrl = if (currentReq.getURI.isAbsolute) currentReq.getURI.toString else (currentHost.toURI + currentReq.getURI)
    currentUrl
  }
}

Tags flattr, scala

Tech

SOGo, GOSa, Postfix, Amavis, Dovecot auf Ubuntu Server 12.04

Post author By Markus Heberling
Post date 2012-05-30
18 Comments on SOGo, GOSa, Postfix, Amavis, Dovecot auf Ubuntu Server 12.04

Ich habe kürzlich mein Mailsystem auf LDAP umgestellt und dann auch gleich noch SOGo installiert um CalDAV und CardDAV zu haben. Hauptsächlich zur Nachvollziehbarkeit für mich selbst sind hier die Schritte, die ich auf einem frisch installierten Ubuntu 12.04 durchgeführt habe. Vielleicht hilft es ja auch jemand anderem. Ich hoffe es ist alles komplett. Wenn das mal jemand nachgemacht hat, würde ich mich über einen Kommentar freuen. Auch für Verbesserungsvorschläge der Konfiguration wäre ich dankbar.

Alle Shell eingaben sind als root durchzuführen, wenn nicht anders angegeben. Als hostname verwende ich hier mail.example.com. Die LDAP Base ist dc=example,dc=com.

Zunächst habe ich erstmal etckeeper installiert, da ich meine Konfiguratinsdateien gerne unter Versionsverwaltung habe:

apt-get install etckeeper git
#/etc/etckeeper/etckeeper.conf editieren
etckeeper init
etckeeper commit

diff -u etckeeper.conf.orig etckeeper.conf
--- etckeeper.conf.orig	2012-05-30 21:40:37.536487058 +0200
+++ etckeeper.conf	2012-05-21 16:52:50.197848685 +0200
@@ -1,7 +1,7 @@
 # The VCS to use.
 #VCS="hg"
-#VCS="git"
-VCS="bzr"
+VCS="git"
+#VCS="bzr"
 #VCS="darcs"

 # Options passed to git commit when run by etckeeper.

GOsa wird zur Verwaltung des LDAP Systems genutzt.

apt-get install slapd gosa-schema gosa gosa-plugin-systems gosa-plugin-mail ldap-utils

Ich bevorzuge das rfc2307bis Schema für LDAP-Gruppen. Wer das nis Schema benutzen möchte, kann den nächsten Schritt weglassen.

service slapd stop
rm /etc/ldap/slapd.d/cn=config/cn=schema/cn={2}nis.ldif
cp /etc/ldap/schema/gosa/rfc2307bis.ldif /etc/ldap/slapd.d/cn=config/cn=schema/cn={2}rfc2307bis.ldif
#rfc2307bis.ldif editieren siehe diff
service slapd start

--- ldap/schema/gosa/rfc2307bis.ldif	2011-04-11 13:41:29.000000000 +0200
+++ ldap/slapd.d/cn=config/cn=schema/cn={2}rfc2307bis.ldif	2012-05-23 14:33:14.664485332 +0200
@@ -1,7 +1,7 @@
 #
 ################################################################################
 #
-dn: cn=rfc2307bis,cn=schema,cn=config
+dn: cn=rfc2307bis
 objectClass: olcSchemaConfig
 cn: rfc2307bis
 #

Hier geht es für alle weiter. Es werden die von GOsa benötigten Schemata zum LDAP hinzugefügt:

ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/gosa/gosystem.ldif
ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/gosa/gofon.ldif
ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/gosa/goto.ldif
ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/gosa/goserver.ldif
ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/gosa/samba3.ldif
ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/gosa/gosa-samba3.ldif

#index setzen
ldapmodify -Y EXTERNAL -H ldapi:///
dn: olcDatabase={1}hdb,cn=config
add: olcDbIndex
olcDbIndex: uid eq
olcDbIndex: mail eq
olcDbIndex: cn eq
olcDbIndex: gosaMailAlternateAddress eq
olcDbIndex: postfixMyDestinations eq

Auf http://mail.example.com/gosa den Anweisungen folgen. Je nachdem ob man die rfc2307bis Gruppen im LDAP aktiviert hat, oder nicht muss die entsprechende Einstellung im GOsa Setup gewählt werden.

Nachdem man die generierte GOsa Konfiguration gespeichert hat, muss zunächst ein neues System in GOsa angelegt werden:

In diesem System kann nun ein neuer Mailserver angelegt werden:

Hierbei ist es wichtig, dass unter dem Punkt “Domänen, für die Mail angenommen wird” der Hostname des Servers eingetragen wird. In diesem Fall also mail.example.com.

Als nächstes werden postfix, dovecot und amavisd-new installiert. pyzor und razor verbessern die SPAM-Erkennungsrate, können aber weggelassen werden, wenn man das möchte. clamav-unofficial-sigs sind SPAM-Signaturen für clamav. mutt ist optional, aber hilfreich beim testen. phpldapadmin ist auch optimal, aber wenn man mal schnell in der LDAP-Datenbank rumeditieren will, ist es ganz hilfreich.

 apt-get install postfix postfix-ldap postfix-policyd-spf-python dovecot-ldap dovecot-imapd dovecot-pop3d dovecot-sieve dovecot-managesieved amavisd-new-postfix pyzor razor clamav-unofficial-sigs mutt phpldapadmin

#alte exim configs entfernen
apt-get --purge remove exim4 exim4-base exim4-config exim4-daemon-light

#damit clamav auf die von amavis bereitgestellten Mails zugreifen darf
adduser clamav amavis

#Mail wird als User vmail unter /var/lib/vmail gespeichert
groupadd -g 800 vmail
useradd -m -u 800 -g 800 -d /var/lib/vmail -s /bin/false vmail

Das auth_username_format wird geändert, damit man sich mit der uid anmelden kann. Ausserdem muss die system Konfiguartion durch die ldap Konfiguration ersetzt werden.

diff --git a/dovecot/conf.d/10-auth.conf b/dovecot/conf.d/10-auth.conf
index d5d2a45..87a9137 100644
--- a/dovecot/conf.d/10-auth.conf
+++ b/dovecot/conf.d/10-auth.conf
@@ -47,7 +47,7 @@
 # the standard variables here, eg. %Lu would lowercase the username, %n would
 # drop away the domain if it was given, or "%n-AT-%d" would change the '@' into
 # "-AT-". This translation is done after auth_username_translation changes.
-#auth_username_format =
+auth_username_format = %n

 # If you want to allow master users to log in by specifying the master
 # username within the normal username string (ie. not using SASL mechanism's
@@ -118,9 +118,9 @@ auth_mechanisms = plain
 #!include auth-deny.conf.ext
 #!include auth-master.conf.ext

-!include auth-system.conf.ext
+#!include auth-system.conf.ext
 #!include auth-sql.conf.ext
-#!include auth-ldap.conf.ext
+!include auth-ldap.conf.ext
 #!include auth-passwdfile.conf.ext
 #!include auth-checkpassword.conf.ext
 #!include auth-vpopmail.conf.ext

In /var/lib/vmail/mail werden die Postfächer der Benutzer gespeichert. Unter /var/lib/vmail/home werden sieve scripts gespeichert.

diff --git a/dovecot/conf.d/10-mail.conf b/dovecot/conf.d/10-mail.conf
index 9c05710..f51402f 100644
--- a/dovecot/conf.d/10-mail.conf
+++ b/dovecot/conf.d/10-mail.conf
@@ -28,6 +28,8 @@
 #
 #
 #mail_location =
+mail_location = maildir:/var/lib/vmail/mail/%u
+mail_home = /var/lib/vmail/home/%u

 # If you need to set multiple mailbox locations or want to change default
 # namespace settings, you can do it by defining namespace sections.
@@ -108,6 +110,8 @@
 # or names.
 #mail_uid =
 #mail_gid =
+mail_uid = vmail
+mail_gid = vmail

 # Group to enable temporarily for privileged operations. Currently this is
 # used only with INBOX when either its initial creation or dotlocking fails.

Die Auth-Listener von Dovecot müssen die richtigen Berechtigungen haben:

diff --git a/dovecot/conf.d/10-master.conf b/dovecot/conf.d/10-master.conf
index f0e79ab..96ec2e5 100644
--- a/dovecot/conf.d/10-master.conf
+++ b/dovecot/conf.d/10-master.conf
@@ -81,13 +81,18 @@ service auth {
   unix_listener auth-userdb {
     #mode = 0600
     #user =
-    #group =
+    #group =
+    user = vmail
+    group = vmail
   }

   # Postfix smtp-auth
-  #unix_listener /var/spool/postfix/private/auth {
+  unix_listener /var/spool/postfix/private/auth {
   #  mode = 0666
-  #}
+    mode = 0660
+    user = postfix
+    group = postfix
+  }

   # Auth process is run as this user.
   #user = $default_internal_user

Dovecot LDAP Konfiguration:

diff --git a/dovecot/dovecot-ldap.conf.ext b/dovecot/dovecot-ldap.conf.ext
index 7dcc748..21e4204 100644
--- a/dovecot/dovecot-ldap.conf.ext
+++ b/dovecot/dovecot-ldap.conf.ext
@@ -14,7 +14,7 @@
 #        by * none

 # Space separated list of LDAP hosts to use. host:port is allowed too.
-#hosts =
+hosts = localhost

 # LDAP URIs to use. You can use this instead of hosts list. Note that this
 # setting isn't supported by all LDAP libraries.
@@ -22,10 +22,10 @@

 # Distinguished Name - the username used to login to the LDAP server.
 # Leave it commented out to bind anonymously (useful with auth_bind=yes).
-#dn = 
+dn = cn=admin,dc=example,dc=com

 # Password for LDAP server, if dn is specified.
-#dnpass = 
+dnpass = LDAP_PASSWORD

 # Use SASL binding instead of the simple binding. Note that this changes
 # ldap_version automatically to be 3 if it's lower. Also note that SASL binds
@@ -64,7 +64,7 @@
 # The pass_filter is used to find the DN for the user. Note that the pass_attrs
 # is still used, only the password field is ignored in it. Before doing any
 # search, the binding is switched back to the default DN.
-#auth_bind = no
+auth_bind = yes

 # If authentication binding is used, you can save one LDAP request per login
 # if users' DN can be specified with a common template. The template can use
@@ -87,7 +87,7 @@

 # LDAP base. %variables can be used here.
 # For example: dc=mail, dc=example, dc=org
-base =
+base = ou=people,dc=example,dc=com

 # Dereference: never, searching, finding, always
 #deref = never
@@ -112,6 +112,7 @@ base =
 #   %n - user part in user@domain, same as %u if there's no domain
 #   %d - domain part in user@domain, empty if user there's no domain
 #user_filter = (&(objectClass=posixAccount)(uid=%u))
+user_filter = (&(objectClass=gosaMailAccount)(uid=%n))

 # Password checking attributes:
 #  user: Virtual user name (user@domain), if you wish to change the
@@ -120,6 +121,7 @@ base =
 # There are also other special fields which can be returned, see
 # http://wiki2.dovecot.org/PasswordDatabase/ExtraFields
 #pass_attrs = uid=user,userPassword=password
+pass_attrs = uid=user,userPassword=password

 # If you wish to avoid two LDAP lookups (passdb + userdb), you can use
 # userdb prefetch instead of userdb ldap in dovecot.conf. In that case you'll
@@ -130,10 +132,11 @@ base =

 # Filter for password lookups
 #pass_filter = (&(objectClass=posixAccount)(uid=%u))
+pass_filter = (&(objectClass=gosaMailAccount)(|(uid=%u)(mail=%u)))

 # Attributes and filter to get a list of all users
-#iterate_attrs = uid=user
-#iterate_filter = (objectClass=posixAccount)
+iterate_attrs = uid=user
+iterate_filter = (objectClass=gosaMailAccount)

 # Default password scheme. "{scheme}" before password overrides this.
 # List of supported schemes is in: http://wiki2.dovecot.org/Authentication

Mailaddressen zu alias auflösen:

diff --git a/postfix/ldap-virtual_alias_maps.cf b/postfix/ldap-virtual_alias_maps.cf
new file mode 100644
index 0000000..4f5362c
--- /dev/null
+++ b/postfix/ldap-virtual_alias_maps.cf
@@ -0,0 +1,8 @@
+server_host = ldap://localhost
+search_base = ou=people,dc=example,dc=com
+bind = no
+
+query_filter = (&(objectclass=gosaMailAccount)(|(|(mail=%s)(gosaMailAlternateAddress=%s))(uid=%s)))
+result_attribute = uid
+#domain = domain.tld
+

LDAP-Gruppen auflösen:

diff --git a/postfix/ldap-virtual_alias_maps_groups.cf b/postfix/ldap-virtual_alias_maps_groups.cf
new file mode 100644
index 0000000..8b0ca10
--- /dev/null
+++ b/postfix/ldap-virtual_alias_maps_groups.cf
@@ -0,0 +1,8 @@
+server_host = ldap://localhost
+search_base = ou=groups,dc=example,dc=com
+bind = no
+
+query_filter = (&(objectclass=gosaMailAccount)(|(mail=%s)(gosaMailAlternateAddress=%s)))
+result_attribute = memberUid, gosaMailForwardingAddress
+#domain = domain.tld
+

LDAP Abfrage um die Domains herauszufinden, für die wir Mail akzeptieren:

diff --git a/postfix/ldap-virtual_mailbox_domains.cf b/postfix/ldap-virtual_mailbox_domains.cf
new file mode 100644
index 0000000..fe94808
--- /dev/null
+++ b/postfix/ldap-virtual_mailbox_domains.cf
@@ -0,0 +1,7 @@
+server_host = ldap://localhost
+search_base = ou=systems,dc=example,dc=com
+bind = no
+
+query_filter = (|(postfixMyDestinations=%s))
+result_attribute = postfixMyDestinations
+

LDAP Abfrage um den Mailboxnamen zu einer Mailaddresse herauszufinden:

diff --git a/postfix/ldap-virtual_mailbox_maps.cf b/postfix/ldap-virtual_mailbox_maps.cf
new file mode 100644
index 0000000..8b8322a
--- /dev/null
+++ b/postfix/ldap-virtual_mailbox_maps.cf
@@ -0,0 +1,7 @@
+server_host = ldap://localhost
+search_base = ou=people,dc=example,dc=com
+bind = no
+
+query_filter = (&(objectclass=gosaMailAccount)(|(mail=%s)(gosaMailAlternateAddress=%s)))
+result_attribute = uid
+

Die LDAP Abfragen dem Postfix bekannt machen und Dovecot als Delivery und SASL Autentifizeirung konfigurieren:

diff --git a/postfix/main.cf b/postfix/main.cf
index c1acd07..a5b3c34 100644
--- a/postfix/main.cf
+++ b/postfix/main.cf
@@ -31,12 +31,33 @@ myhostname = mail.example.com
 alias_maps = hash:/etc/aliases
 alias_database = hash:/etc/aliases
 myorigin = /etc/mailname
-mydestination = mail.example.com, localhost.example.com, , localhost
+mydestination = localhost.example.com, , localhost
 relayhost =
 mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
 mailbox_size_limit = 0
 recipient_delimiter = +
 inet_interfaces = all
+inet_protocols = all
+
+smtpd_sasl_type = dovecot
+
+# Can be an absolute path, or relative to $queue_directory
+# Debian/Ubuntu users: Postfix is setup by default to run chrooted, so it is best to leave it as-is below
+smtpd_sasl_path = private/auth
+
+# and the common settings to enable SASL:
+smtpd_sasl_auth_enable = yes
+smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination, check_policy_service unix:private/policy-spf
+
 content_filter = smtp-amavis:[localhost]:10024
 policy-spf_time_limit = 3600s
-smtpd_recipient_restrictions = permit_mynetworks, reject_unauth_destination, check_policy_service unix:private/policy-spf
+
+virtual_mailbox_maps = ldap:/etc/postfix/ldap-virtual_mailbox_maps.cf
+virtual_mailbox_domains = ldap:/etc/postfix/ldap-virtual_mailbox_domains.cf
+virtual_alias_maps = ldap:/etc/postfix/ldap-virtual_alias_maps_groups.cf, ldap:/etc/postfix/ldap-virtual_alias_maps.cf
+
+virtual_gid_maps = static:vmail
+virtual_uid_maps = static:vmail
+virtual_transport = dovecot
+dovecot_destination_recipient_limit = 1
+

Dovecot als delivery agent dem Postfix bekannt machen:

diff --git a/postfix/master.cf b/postfix/master.cf
index 35512c9..746d5b2 100644
--- a/postfix/master.cf
+++ b/postfix/master.cf
@@ -110,6 +110,9 @@ scalemail-backend unix      -       n       n       -       2       pipe
 mailman   unix  -       n       n       -       -       pipe
   flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
   ${nexthop} ${user}
+#dovecot
+dovecot   unix  -       n       n       -       -       pipe
+    flags=DRhu user=vmail:vmail argv=/usr/lib/dovecot/deliver -d ${recipient} -f ${sender}

 # ==========================================================================
 # service type  private unpriv  chroot  wakeup  maxproc command + args

Virenscanner und SPAM-Checker im amavis aktivieren:

diff --git a/amavis/conf.d/15-content_filter_mode b/amavis/conf.d/15-content_filter_mode
index 57c62c8..7b1a2ac 100644
--- a/amavis/conf.d/15-content_filter_mode
+++ b/amavis/conf.d/15-content_filter_mode
@@ -10,8 +10,8 @@ use strict;
 # If You wish to enable it, please uncomment the following lines:

-#@bypass_virus_checks_maps = (
-#   \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);
+@bypass_virus_checks_maps = (
+   \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);

 #
@@ -21,7 +21,7 @@ use strict;
 # If You wish to enable it, please uncomment the following lines:

-#@bypass_spam_checks_maps = (
-#   \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re);
+@bypass_spam_checks_maps = (
+   \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re);

 1;  # ensure a defined return

Sieve support konfigurieren:

diff --git a/dovecot/conf.d/15-lda.conf b/dovecot/conf.d/15-lda.conf
index 609023a..4b7a2de 100644
--- a/dovecot/conf.d/15-lda.conf
+++ b/dovecot/conf.d/15-lda.conf
@@ -44,5 +44,5 @@

 protocol lda {
   # Space separated list of plugins to load (default is global mail_plugins).
-  #mail_plugins = $mail_plugins
+  mail_plugins = $mail_plugins sieve
 }

SOGo nutzt den Port 2000 für Sieve:

diff --git a/dovecot/conf.d/20-managesieve.conf b/dovecot/conf.d/20-managesieve.conf
index a73a417..e5e17cf 100644
--- a/dovecot/conf.d/20-managesieve.conf
+++ b/dovecot/conf.d/20-managesieve.conf
@@ -9,9 +9,9 @@ service managesieve-login {
   #  port = 4190
   #}

-  #inet_listener sieve_deprecated {
-  #  port = 2000
-  #}
+  inet_listener sieve_deprecated {
+    port = 2000
+  }

   # Number of connections to handle before starting a new process. Typically
   # the only useful values are 0 (unlimited) or 1. 1 is more secure, but 0

SOGo benötigt für einige Regeln imapflags:

diff --git a/dovecot/conf.d/90-sieve.conf b/dovecot/conf.d/90-sieve.conf
index 516ac46..79de623 100644
--- a/dovecot/conf.d/90-sieve.conf
+++ b/dovecot/conf.d/90-sieve.conf
@@ -45,6 +45,7 @@ plugin {
   # deprecated imapflags extension in addition to all extensions thatwere
   # already enabled by default.
   #sieve_extensions = +notify +imapflags
+  sieve_extensions = +imapflags

   # The Pigeonhole Sieve interpreter can have plugins of its own. Using this
   # setting, the used plugins can be specified. Check the Dovecot wiki

An dieser stelle sollte das Mail-System funktionieren.

Zunächst muss das SOGo repository in apt eingebunden werden. Ich habe den nightly build von SOGo 2.0 genommen, die Konfiguration sollte aber auch für SOGo 1.x passen.

/etc/apt/sources.list.d/sogo.list:

deb http://inverse.ca/ubuntu-nightly precise precise
#deb http://inverse.ca/ubuntu precise precise

Apache kann automatisch konfiguriert werden. Für mysql sollte ein Passwort gesetzt werden. phpmyadmin ist optional, aber hilfreich wenn man mal in der Datenbank rumfuhrwerken will. Ich werde spöter phpmyadmin dazu benutzen die SOGo Datenbank anzulegen, aber das kann man natürich auch über die shell machen.

apt-key adv --keyserver keys.gnupg.net --recv-key 0x810273C4
apt-get update
apt-get install sogo mysql-server phpmyadmin memcached

Es müssen noch ein paar module für den apache aktiviert werden:

a2enmod proxy
a2enmod proxy_http
a2enmod rewrite
a2enmod substitute

SOGo legt seine config im home directory des sogo Benutzers an. Das liegt allerdings nicht unter /etc und wird daher nicht vom etckeeper verwaltet. Das ist doof. Deshalb verschiebe ich das home directory nach /etc/sogo:

/etc/init.d/sogo stop
usermod -m -d /etc/sogo sogo

Jetzt wird auch die SOGo Datenbank angelegt. Ich habe phpmyadmin dazu benutzt:

Und hier ist meine SOGo Konfigurationsdatei.:

diff --git a/sogo/GNUstep/Defaults/.GNUstepDefaults b/sogo/GNUstep/Defaults/.GNUstepDefaults
index 8c01f2e..e2f0438 100644
--- a/sogo/GNUstep/Defaults/.GNUstepDefaults
+++ b/sogo/GNUstep/Defaults/.GNUstepDefaults
@@ -7,6 +7,102 @@
     </dict>
     <key>sogod</key>
     <dict>
+        <key>OCSFolderInfoURL</key>
+        <string>mysql://sogo:MYSQL_PASSWORD@localhost:3306/sogo/sogo_folder_info</string>
+        <key>OCSSessionsFolderURL</key>
+        <string>mysql://sogo:MYSQL_PASSWORD@localhost:3306/sogo/sogo_sessions_folder</string>
+        <key>SOGoACLsSendEMailNotifications</key>
+        <string>YES</string>
+        <key>SOGoAppointmentSendEMailNotifications</key>
+        <string>YES</string>
+        <key>SOGoAuthenticationMethod</key>
+        <string>LDAP</string>
+        <key>SOGoPasswordChangeEnabled</key>
+        <string>YES</string>
+        <key>SOGoEnablePublicAccess</key>
+        <string>YES</string>
+        <key>SOGoFirstDayOfWeek</key>
+        <string>1</string>
+        <key>SOGoFirstWeekOfYear</key>
+        <string>First4DayWeek</string>
+        <key>SOGoFoldersSendEMailNotifications</key>
+        <string>YES</string>
+        <key>SOGoLanguage</key>
+        <string>German</string>
+        <key>SOGoMailDomain</key>
+        <string>example.com</string>
+        <key>SOGoMailMessageCheck</key>
+        <string>every_minute</string>
+        <key>SOGoProfileURL</key>
+        <string>mysql://sogo:MYSQL_PASSWORD@localhost:3306/sogo/sogo_user_profile</string>
+        <key>SOGoSieveScriptsEnabled</key>
+        <string>YES</string>
+        <key>SOGoSieveServer</key>
+        <string>sieve://localhost:2000</string>
+        <key>SOGoTimeZone</key>
+        <string>Europe/Berlin</string>
+        <key>SOGoUserSources</key>
+        <array>
+            <dict>
+                <key>CNFieldName</key>
+                <string>cn</string>
+                <key>IMAPHostFieldName</key>
+                <string>gosaMailServer</string>
+                <key>UIDFieldName</key>
+                <string>uid</string>
+                <key>IMAPLoginFieldName</key>
+                <string>uid</string>
+                <key>baseDN</key>
+                <string>ou=people,dc=example,dc=com</string>
+                <key>bindDN</key>
+                <string>cn=admin,dc=example,dc=com</string>
+                <key>bindFields</key>
+                <string>uid</string>
+                <key>bindPassword</key>
+                <string>LDAP_PASSWORD</string>
+                <key>canAuthenticate</key>
+                <string>YES</string>
+                <key>displayName</key>
+                <string>Shared Addresses</string>
+                <key>hostname</key>
+                <string>localhost</string>
+                <key>id</key>
+                <string>public</string>
+                <key>isAddressBook</key>
+                <string>YES</string>
+                <key>port</key>
+                <string>389</string>
+                <key>type</key>
+                <string>ldap</string>
+            </dict>
+           <dict>
+                <key>CNFieldName</key>
+                <string>cn</string>
+                <key>IDFieldName</key>
+                <string>cn</string>
+                <key>UIDFieldName</key>
+                <string>cn</string>
+                <key>baseDN</key>
+                <string>ou=groups,dc=example,dc=com</string>
+                <key>bindDN</key>
+                <string>cn=admin,dc=example,dc=com</string>
+                <key>bindPassword</key>
+                <string>LDAP_PASSWORD</string>
+                <key>canAuthenticate</key>
+                <string>YES</string>
+                <key>displayName</key>
+                <string>Groups</string>
+                <key>hostname</key>
+                <string>127.0.0.1</string>
+                <key>id</key>
+                <string>t-srv_groups</string>
+                <key>isAddressBook</key>
+                <string>YES</string>
+                <key>port</key>
+                <string>389</string>
+            </dict>
+        </array>
     </dict>
 </dict>
-</plist>
\ No newline at end of file
+</plist>
+

Die Apache Konfiguration für SOGo muss noch angepasst werden. Den Teil mit der Erstellung von SSL Zertifikaten habe ich hier mal rausgelassen.

diff --git a/apache2/conf.d/SOGo.conf b/apache2/conf.d/SOGo.conf
index 040814e..64ed4a2 100644
--- a/apache2/conf.d/SOGo.conf
+++ b/apache2/conf.d/SOGo.conf
@@ -30,6 +30,14 @@ ProxyRequests Off
 SetEnv proxy-nokeepalive 1
 ProxyPreserveHost On

+#timezone fix
+RequestHeader unset Accept-Encoding
+
+AddOutputFilterByType SUBSTITUTE text/calendar text/xml
+Substitute 's|^RRULE:([A-Z].+[A-Za-z0-9]);?FREQ=([A-Za-z]*)(;?.*)$|RRULE:FREQ=$2;$1$3|' 
+#Substitute 's|RRULE:BYDAY=-1SU;FREQ=YEARLY;BYMONTH=10|RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10|n'
+#Substitute 's|RRULE:BYDAY=-1SU;FREQ=YEARLY;BYMONTH=3|RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=3|n'
+
 # When using CAS, you should uncomment this and install cas-proxy-validate.py
 # in /usr/lib/cgi-bin to reduce server overloading
 #
@@ -65,3 +73,55 @@ ProxyPass /SOGo http://127.0.0.1:20000/SOGo retry=0
 # header of emails.
 RewriteEngine On
 RewriteRule ^/SOGo/(.*)$ /SOGo/$1 [env=REMOTE_HOST:%{REMOTE_ADDR},PT]
+
+
+Listen 8843
+NameVirtualHost 0.0.0.0:8843
+<VirtualHost 0.0.0.0:8843>
+ ServerName mail.example.com
+ SSLEngine On
+
+        #mitigate beast attack
+        SSLHonorCipherOrder On
+        SSLCipherSuite ECDHE-RSA-AES128-SHA256:AES128-GCM-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH
+
+        #   A self-signed (snakeoil) certificate can be created by installing
+        #   the ssl-cert package. See
+        #   /usr/share/doc/apache2.2-common/README.Debian.gz for more info.
+        #   If both key and certificate are stored in the same file, only the
+        #   SSLCertificateFile directive is needed.
+        SSLCertificateFile    /etc/ssl/private/mail.example.com.pem
+        SSLCertificateKeyFile /etc/ssl/private/mail.example.com.pem
+
+        #   Server Certificate Chain:
+        #   Point SSLCertificateChainFile at a file containing the
+        #   concatenation of PEM encoded CA certificates which form the
+        #   certificate chain for the server certificate. Alternatively
+        #   the referenced file can be the same as SSLCertificateFile
+        #   when the CA certificates are directly appended to the server
+        #   certificate for convinience.
+        SSLCertificateChainFile /etc/ssl/private/mail.example.com.pem
+
+ProxyRequests Off
+SetEnv proxy-nokeepalive 1
+ProxyPreserveHost On
+
+ProxyPassInterpolateEnv On
+#for CardDAV
+ProxyPass /principals http://127.0.0.1:20000/SOGo/dav/ interpolate
+ProxyPass /SOGo/dav/ http://127.0.0.1:20000/SOGo/dav/ interpolate
+ProxyPass / http://127.0.0.1:20000/SOGo/dav/ interpolate
+
+<Proxy http://127.0.0.1:20000>
+ RequestHeader set "x-webobjects-server-port" "8843"
+ RequestHeader set "x-webobjects-server-name" "mail.example.com:8843"
+ RequestHeader set "x-webobjects-server-url" "https://mail.example.com:8843"
+ RequestHeader set "x-webobjects-server-protocol" "HTTP/1.0"
+ RequestHeader set "x-webobjects-remote-host" "127.0.0.1"
+ AddDefaultCharset UTF-8
+ Order allow,deny
+ Allow from all
+</Proxy>
+
+</VirtualHost>
+

Jetzt muss noch SOGo gestartet und der Apache neu gestartet werden:

service sogo start
service apache2 restart

Ich hoffe mal, dass ich alle relevanten Einstellungen beschrieben habe. Kommentare gerne in den Kommentaren. 🙂

Tags dovecot, gosa, ldap, mail, postfix, sogo, ubuntu

Tech

MouseOver mit Selenium-WebDriver

Post author By Markus Heberling
Post date 2012-03-16
No Comments on MouseOver mit Selenium-WebDriver

Manchmal™ möchte man Webseiten testen, die ein Menü einblenden, wenn man mit der Maus über einem Element ist. Folgender Code für Selenium WebDriver macht genau das:

protected void mouseOver(WebElement element) {
    String code = "var fireOnThis = arguments[0];"
                + "var evObj = document.createEvent('MouseEvents');"
                + "evObj.initEvent( 'mouseover', true, true );"
                + "fireOnThis.dispatchEvent(evObj);";
    ((JavascriptExecutor) driver).executeScript(code, element);
}

Tags Java, Selenium

Tech

Monitoring von ksm mit munin

Post author By Markus Heberling
Post date 2011-04-13
No Comments on Monitoring von ksm mit munin

Ich habe auf meinem Server kvm mit Kernel Samepage Merging am laufen. Das wollte ich in munin monitoren, aber da gab es bisher kein Plugin für.

Ich hab da mal eins zusammengebastelt:

ksm_pages_absolute-day — KSM Pages Absolute

ksm_pages_relative-day — KSM Pages Relative

Für Ubuntu gibt’s das auch in meinem PPA. Im Paket munin-plugins-contrib sind im Moment (v0.3) folgende Plugins enthalten:

Tags ksm, munin, ubuntu

Tech

LWUIT on MicroEmulator on iPhone

Post author By Markus Heberling
Post date 2009-03-10
8 Comments on LWUIT on MicroEmulator on iPhone

I tried to run the LWUIT Demo with my MicroEmulator iPhone port. Here are some screenshots:

First Try. Got the clipping slightly wrong…

After a quick fix, clipping correct, but no text…

Changing to another theme…

… the text is there!

Menu is workin.

Setting some dialog properties, and…

… displaying it.

Some more screenshots…

Performance is slow as hell, not all graphics are rendered properly, but its working at least.

Tags iPhone, J2ME, Java, LWUIT, MicroEmulator

Tech

MicroEmulator on iPhone

I have started porting microemulator to (jailbroken) iPhones. Here are some first screenshots.

Tags iPhone, J2ME, Java, MicroEmulator

Tech

Using Maven2 to build Cydia Packages for the iPhone

Post author By Markus Heberling
Post date 2008-08-31
1 Comment on Using Maven2 to build Cydia Packages for the iPhone

I am using Maven2 for most Java projects. So I wanted to use it for Phone-Java development, too. This is a little tutorial on how it can be done.

iPhone Cydia Packages are Debian .deb files really and luckily there is the mvn-pkg-plugin that supports creating .deb files with Maven2. Unfortunatly the current version has a bug, so that it won’t work out of the box for creating iPhone packages. So you have to grab the latest svn-snapshot and apply the following patch.

Index: src/main/java/de/tarent/maven/plugins/pkg/map/Parser.java
===================================================================
--- src/main/java/de/tarent/maven/plugins/pkg/map/Parser.java	(revision 132)
+++ src/main/java/de/tarent/maven/plugins/pkg/map/Parser.java	(working copy)
@@ -62,7 +62,7 @@

       if (auxMapDocument != null)
         {
-          s = new State(packageMapDocument);
+          s = new State(auxMapDocument);

           s.nextMatch("package-maps");
           parsePackageMaps(s);

After applying the atch and doing a “mvn install”, we can start with the real work. Download the tutorial files from here. I will explain the contents of the archive later. For now unzip it and do a “mvn package” in its directory. It will compain about some missing dependencies, so grab the missing jars from your iPhone and install them in your Maven reposiory as directed in the error message. After you have done that another “mvn package” shoud produce a .deb file in the target subdirectory. You can upload this to your iphone and install it with dpkg. After a restart of the SpringBoard a new icon HelloMaven should be on your screen, that should start the demo program.

Now some explanation of how it works. The main part is done in the pom.xml. Here is the relavant excerpt form this file:

	de.tarent.maven.plugins
	maven-pkg-plugin
	
		iphoneos
		Arindal Tune Helper
		
			HelloJava
			example
			iphoneos-arm
			/Applications/HelloMaven.app
			
			
				
					Info.plist
				
				
					icon.png
				
				
					Starter
				
			
			postInst.sh
			preRm.sh
		
		file:${basedir}/pm-iphoneos.xml
		
	
	
		
			package
			
				pkg

The first important part is the defaultDistro tag. Here we tell the plugin, that we want to build a package for the iPhone. Unfortunatly there are no default iPhone mappings contained in the plugin. That is why we have the auxPackageMapURL tag, that tells the plugin where to look for the Maven-to-Cydia-Package-Mapping. Most other things should be pretty self explaining.

The files in the dataFiles Section have to reside in src/main/auxfiles icon.png and Info.plist should be pretty clear. The Starter entry needs some explanation though. The plugin automatically generates a start script in /usr/bin. This is good enough for console applications, but if you want to start a GUI-App some more work is needed. The Starter script sets the Java executable path to /Applications/Appname.app/Java and calls the generated script after that. For this to work the starter Script has to get executable permissions and jamvm has to be linked to /Applications/Appname.app/Java. This is done in the postInst.sh script. For more info about those scripts and what you can do there refer to the Debian documentation.

This should be enough to get you started, if cou have questions, please feel free to post them in the comments.

Tags iPhone, Java, Maven

Tech

iPhone Java First Try

Made my first tries with Java on a jailbreaked iPhone 3G. Here is a screenshot of some drawing routines. Its reacting on touch events, too. Will write more about it later.

So die ersten Versuche mit iPhone Java Entwicklung sind gemacht. Hier mal ein Screenshot von ein paar Grafikausgaben. Reagiert auch schon auf Touch Events. Spaeter mehr davon.

Tags iPhone, Java

Tech

Parse Command Line Options with annotations

Post author By Markus Heberling
Post date 2008-07-10
No Comments on Parse Command Line Options with annotations

Panzi has written a very nice Lib to parse command line options in Java. You can just add Annotations to your Java fields and they will be filled automagically with the command line values.

Eine sehr nützliche Lib um Kommandozeilen in Java zu Parsen hat Panzi in seinem Blog beschrieben. Dazu werden die Felder im Java Code einfach mit bestimmten Annotations versehen und die Lib kümmert sich um das Parsing und schreibt die Werte direkt in die Felder.

Tags Java

Tech

WordPress on Ubuntu Intrepid

Post author By Markus Heberling
Post date 2008-07-07
No Comments on WordPress on Ubuntu Intrepid

Es ist sehr leicht WordPress 2.5 unter Ubuntu Intrepid zu installieren. Damit hat man dann auch sofort Multiblog-Fähigkeiten. Es gibt dazu ein Beispielscript (/usr/share/doc/wordpress/examples/setup-mysql) um neue Instanzen anzulegen. Allerdings ist das von Debian kopiert und einige Pfadangaben passen nicht für Ubuntu. Das folgende diff behebt das. Getestet mit wordpress-2.5.1-2ubuntu1.

Its pretty easy to install WordPress 2.5 on Ubuntu Intrepid. The Ubuntu version comes with Multiblog capabilities included. There is an example script (/usr/share/doc/wordpress/examples/setup-mysql) to create new instances of WordPress Blogs. Unfortunatly the script was copied from Debian and some paths are wrong for Ubuntu. The script below fixes that issue. After that it worked flawlessly for me with wordpress-2.5.1-2ubuntu1.

82,83c82,83
< [ -d /srv/www ] || mkdir -p /srv/www
 [ -d /var/www ] || mkdir -p /var/www
> ln -s /usr/share/wordpress /var/www/$DOMAIN
149c149
 UPLOAD="/var/www/wp-uploads/$DOMAIN"
181a182
>

Tags intrepid, ubuntu, Wordpress