this setting was not used. the baud rate for the SDM is set to 9600 in
the source code. until the baud rate being customizable is actually
required by somebody, we remove the setting altogether.
"powertotal" is always published and it is published by the base class
directly. other values are still published by the derived classes, but
use a base class method, which takes care that a common base topic is
used in particular.
instead of iterating a map with subscriptions, we now bind the target
variable to the callback, which is executed once a message is arrived.
this way, the target variable is already linked to the respective topic
when the callback is executed.
lock the mutex when writing the variable, as the MQTT callback is
executed in a different context (MQTT task) than the main loop task,
which otherwise accesses the variables.
it is important to separate the capabilities of each power meter
provider into their own class/source file, as the providers work
fundamentally different and their implementations must not be
intermangled, which made maintenance and improvements a nightmare
in the past.
this only changes line endings. inspect this commit with command `git
show <commit-sha> --ignore-space-at-eol` and it will tell you that the
commit appears to be "empty" (since all changes are whitespace changes
near the end of a line, which are ignored in that git show command).
the files to be changed were found and updated using this command:
find lib src include webapp/src -type f | \
xargs grep --binary-files=without-match --files-with-matches \
$(printf '\r\n') | xargs dos2unix
the following files were restored afterwards, as they are using CRLF
line endings in the upstream as well:
- lib/CMT2300a/cmt2300a_defs.h
- lib/README
- include/README
the "serial console" over USB would be garbled badly after switching to
"platform = espressif32@6.7.0". this did not happen to the upstream
version of MessageOutput. we used Serial.flush(), which seemed to be
good in the respective context. however, the changes in
github.com/espressif/arduino-esp32/pull/9462
made flush() detrimental. we remove the use of flush(), as it seems not
to be required (in particular, the upstream project does not use it).
broke in be41e6b9. was unusable as the complete type of DummySerial has
to be available in the JK BMS controller header when defining the
unique_ptr managing the dummy instance. this problem is solved by moving
the whole dummy class into its own header.
instead of hard-coding the use of hardware UART 2, the SDM power meter
instance now asks for a free hardware serial port to use and
instanciates the respective HardwareSerial object using said port.
get rid of particular compile-time designations by UART index. just hand
out the next free index of hardware UARTs, or indicate that none is
available any more.
use names as keys to register and free UARTs.
only on ESP32-S3-USB. this fiddles with the available hardware UARTs to
make it possible to use a third Victron MPPT. if three MPPTs are defined
int the pin mapping, you will not be able to use the SmartShunt and JK
BMS battery interfaces.
note that using a second MPPT will also conflict with the SDM power
meter, and that conflict is not detected, yet.
this implements accessing array members in an ArduinoJSON object
following the FirebaseJson syntax. the FirebaseJson lib was previously
removed to save flash memory, and logic was implemented to find a JSON
node using the FirebaseJson path syntax, restoring the functionality.
however, array access was not implemented.
this change also addresses leading and trailing and double forward
slashes in the path expression.
moreover, much more expressive error messages are now generated in case
the path could not be resolved.
mbedtls is already integral part of the firmware. use it in favor of
rweather/Crypto library to calculate a sha256 checksum of a string, as
used in the HTTP power meter implementation.
we used this library solely to interpret the answer of an HTTP web
server as JSON and find a particular value using a path expression in
the HTTP power meter implementation.
since we ran out of flash memory on non-S3 ESP32, we need to cut some
corners. removing FirebaseJson is the last low-hanging fruit that we
currently know of. we can get rid of it by using ArduinoJson (which is
already integral part of the firmware) and implementing a custom logic
to extract a value based on a path expression.
other than the FirebaseJson path "finder", the new implementation
only knows how to access sub-keys delimited by a forward slash. in
particular, accessing array members is not supported any more. I am
hoping that this is simply not an issue. if so, we will have users
complaining and we can add this functionality in a later release.
merge upstream tag v24.4.12, resolve conflicts (helgeerbe), fix eslint errors (schlimmchen) and adopt new web api method to save code duplication (schlimmchen).
this allows to use two VE.Direct interfaces, as there is no conflict
regarding HW serial port 2 after making the battery interfaces use
serial port 0 on devices with USB CDC. on those chips HW serial 0 is
free to be used since serial messages are written through the USB
interface directly.
we found that the inverter sometimes stops responding to commands,
especially to the "start producing" command. we now count the number of
consecutive timeouts when trying to send a new limit or power state
commands. after two timeouts were recorded, every additional timeout
will send a restart command to the inverter.
as a last resort, if the counter keeps climbing, the DTU is restarted.
notice that this only targets unresponsive inverters which are
reachable. unreachable inverters are not restarted and do not cause a
DTU reboot. this is important for solar-driven inverters, which are
unreachable during the night. the DPL will not calculate a new limit and
hence the updateInverter() method will do nothing while the target
inverter is unreachable.
publish the timeout counter to MQTT for monitoring purposes.
avoid performing a calculation based on a (slightly) outdated power
meter reading, which was aquired just before the limit was actually
applied by the inverter, but which was received by OpenDTU-OnBattery
after the inverter stats.
without a power meter configured, the DPL now sets the base load as the
inverter limit if the battery charge allows it. it also takes
solar-passthrough into account, i.e., if the battery is in a charge
cycle but the solar output (Victron MPPT) is significant, the solar
power will be used up until the base load. if the battery reaches the
full solar passthrough threshold, the DPL will match the inverter limit
to the MPPT solar output.
on power meter issues (usually a timeout), keep the inverter enabled and
make it produce the configured base load limit if the battery can be
discharged. that should be okay since the base load config value is
expected to be small and a little less than the actual household base
load, i.e., if this amount of power is produced, the household will
consume it in any case and no energy is fed into the grid.
makes the value match its description. since most values in the top part
of the live view are related to the AC side of the system, it makes
sense to use the correct value rather than to change the description.
* process "IL", "AR" and "MON"
* discard "BMV" and (unsolicited) History Data
* simplify isDataValid()
* veMpptStruct, veStruct: new, verbose variable names, including units,
and replace floats (save values with original integer precision)
* comment on rollover situation in isDataValid()
1. makes the DPL use the power generated by all connected charge
controllers for calculations based on solar passthrough.
2. makes the network total DC power appear as "MPPT Total Power" in the
live view at the top.
3. shows the network total DC power in the VE.Direct live data card.
* show charge controller temperature in live view
* send hex requests right after decoding a frame. this seems to have the
best chance of getting an answer to all requests.
* deem 0xFFFFFFFF value of network total DC power as invalid indicator.
neither network state, nor network info, nor network mode seem to
indicate that the charge controller is part of a VE.Smart network. for
that reason, we revert to always querying the network total DC power
value, but testing it for max(uin32_t) value, which seems to indicate
that the charge controller is not part of a VE.Smart network.
* improve (verbose) logging, e.g., use _logId, and print names of
response codes and known registers, always print error messages,
add additional tests to prevent overly verbose messages.
* move hex protocol definitions to VeDirectData.h header
and use enum classes
* define register addresses in enum class
* move values retrieved through hex protocol into main MPPT data struct
* do not send HEX requests if the serial interface cannot send data
* detect whether smart battery sense temperature is available
* web app: make all VE.Direct sub-cards iterable. this makes addind more
values much simpler and saves a bunch of code in the web app.
* make VeDirectFrameHandler state a type-safe enum class
* unindent MPPT controller loop()
* whitespace cleanup
double precision floating point numbers are not needed to handle
VE.Direct values. handling double is implemented in software and hence
*much* more resource intensive.
queue every text event until the frame was checked by it checksum. then
process the data directly into the buffer struct. do not clear the
buffer struct, so it will always include the most recent value of a
particular data point.
if only a single request was made (switch "Individual HTTP requests per
phase" is off), the user could still enable phase 2 and phase 3 config
and configure a respective JSON path. however, the value was never
extracted from the successful HTTP request for phase 1.
closes#637.
users are manipulating the DPL using HTTP POST requests. often they are
requesting the current settings using HTTP GET on the respective route,
then change a particular settings, and send all the data back using HTTP
POST. if they failed to remove the metadata node from the JSON,
OpenDTU-OnBattery would not be able to process the JSON due to its size.
the web app does not submit the metadata.
to avoid problems, the metadata is now split from the configuration
data.
this method calls the overflowed() method on the respective
DynamicJsonDocument and prints a respective message if not all
data could be added to the DynamicJsonDocument.
in case the user defines the respective pins, an MPPT controller will be
setup and the websocket would previously publish its invalid data even
if no data at all was received. this lead to using an invalid index and
to publish a useless card. instead, skip those controllers.
if the connection to a controller breaks, it will always present as a
card in the live view with the respective serial number and the last
known data.
PLLIMIT would not (always) be part of the JSON string transmitted to the
web application and trip it. with the help of the ArduinoJson assistant,
new values to use in the response size calculation were introduced.
also, importantly, a constant offset was added for the DPL status and
the JSON structure sourrounding the MPPTs.
* updating the SoC or value shall also update the general timestamp, as
the latter is defined as "any value changed", which includes SoC and
voltage, of course.
* if the last update is not a valid timestamp at all, the
updateAvailable method must always return false, obviously.
on the "main" live data websocket this can happen if no
OnBattery-specific data is available to sent but the empty
JSON document is still sent, which trips the web application.
publishing null in the battery live data websocket also trips the web
application, which rightfully assumes a valid object if data is received
through the websocket.
determine the amount of controllers actually in use dynamically,
especially to avoid indices which are invalid, causing an error
to be printed, even though the user did not do anything wrong.
in your pin_mapping.json, add a powermeter object like this:
[
{
"name": "My Board",
...
"powermeter": {
"rx": <num>,
"tx": <num>,
"dere": <num>
},
...
}
]
the SML power meter requires the rx pin to be set. the SDM power meter
requires the rx and tx pins are set. the "dere" pin pin is optional and
if set, this pin controls the driver enable and receiver enable pins of
the RS485 transceiver. the SDM library handles this pin.
closes#771.
* remove/comment unused variables to avoid compiler warnings
* cleanups: fix indention and style, make variable private, implement
getters in header and make const.
* optimize message output: respect verbose logging setting. prefix
output with SMA_HM.
* use newly introduced mutex in PowerMeterClass also for SMA HomeManager
* refactor code for readibility, unindent where possible.
the SDM power meter (among others) writes the power consumption of three
phases in multiple steps. this change helps to prevent getTotalPower()
reading intermediate values, e.g., reading a new value for phase 1 but
old values for phase 2 and 3 since phase 2 is currently read.
cache the values, and write them all at once, protected by a mutex,
later.
closes#732.
this changeset refactors the web application's DPL settings view. the
DPL settings can be complex, and they shall be presented in a way that
allows users to comprehend their meaning. irrelevant settings are now
hidden or displayed dynamically based on the influencing settings.
* group SoC thresholds into their own card
* hide battery SoC thresholds if battery disabled. if the user did not
even enable the battery interface, battery SoC values will not be used
for DPL decisions. in that case we completely hide the respective
settings from the DPL admin view. this reduces the amount of settings
for new users and especially users who don't even have a battery in
their setup or have no BMS connected.
* group voltage thresholds and improve label texts
* fix load correction factor unit
* fix header (wording)
* group solar-passthrough settings in new card
* group inverter-related settings
* hide solar passthrough settings if VE.Direct is disabled. closes#662.
* completely disable form if any requirement is not met
* list available inverters by name and type. this makes it much more
convenient to select the right inverter, especially since the order of
the inverters in the web UI is decoupled from their position in the
internal array, which was used to select them previously. care was
taken that old configs select the same inverter after an update.
when editing the DPL settings, the selects an inverter from the newly
created drow-down list, and the respective old inverter is
pre-selected.
* disable form if no inverter is configured (config alert)
* make inverter input selection dynamic. adjust selection to actual
amount of channels for selected inverter. skip selection altogether if
inverter has only one channel, or if it is solar powered.
* web app: wording adjustments
* group meta data into new property and exclude from submission. saves
memory when evaluating the submitted settings.
* hide irrelevant settings if inverter is solar-powered
* move restart hour setting to inverter card. translate setting which
disabled automatic restart.
* simplify "drain strategy" setting into an on/off toggle. care was
taken that existing configs work the same after an upgrade. the
respective drain strategy is translated into the new setting when
reading the config. once the config is written, the new setting is
persisted and the old is not part of the config any more.
* show more configuration hints, depending on actual configuration
* replace inputs by InputElement components where possible
* fix compiler warning in SerialPortManager.cpp: function must not
return void
* clean up and simplify implementation of usesHwPort2()
* make const
* overrides are final
* default implementation returns false
* implement in header, as the implementation is very simple
* rename PortManager to SerialPortManager. as "PortManager" is too
generic, the static instance of the serial port manager is renamed to
"SerialPortManager". the class is therefore renamed to
SerialPortManagerClass, which is in line with other (static) classes
withing OpenDTU(-OnBattery).
* implement separate data ages for MPPT charge controllers
* make sure MPPT data and live data time out
* do not use invalid data of MPPT controlers for calculations
* add :key binding to v-for iterating over MPPT instances
this change adds support for a second Victron MPPT charge controller
using a second serial connection.
* Add device configuration for a second victron mppt
* Update VedirectView for second victron mppt
* Update MqttHandleVedirect for second victron mppt
* Update MqttHandleVedirectHass for second victron mppt
* Handle nonexisting victron controllers with optionals
* Add bool-function to Battery and inherited classes, if uart port 2 is
being used
* Introduced a serial port manager. In order to prevent the battery and
the Victron MPPT to use the same hw serial ports, this class keeps
track of the used ports and their owners.
now that users can tell the DPL that their inverter is not powered by a
battery but powered by solar panels, we shall not restart inverters to
reset the daily yield value, if they are solar powered. these inverters
will reboot every night by themselves.
* fix logic in HomeAssistent handler
* also publish voltage thresholds (not just SoC thresholds)
* do not publish irrelevant thresholds to MQTT. if the inverter is
solar-powered, no thresholds are effectively in use by the DPL and it
therefore makes no sense to publish them to the broker. similarly, if
no battery interface is enabled or the SoC values are set to be
ignored, the SoC thresholds are effectively not in use and will not be
published to the broker.
* make HA auto-discovery expire. this makes auto-dicovered items
disappear from Home Assistent if their value is no longer updated.
changes to settings which cause other thresholds to be relevant will
then be reflected in Home Assistent even if some thresholds are no
longer maintaned in MQTT.
* force HA update when related settings change enabling VE.Direct shall
trigger an update since solar passthrough thresholds become relevant.
similarly, enabling the battery interface makes SoC thresholds become
relevant. there are more settings in the power limiter that also
influence the auto-discoverable items.
* break very long lines
the allocated memory to create the JSON with onBattery-specific totals
for the live view was too little to contain all values, which are sent
regularly.
avoid shutting down the inverter at all if the calculated power limit
falls below the lower power limit or if the power meter value is
outdated. do this only if the inverter is setup to be solar powered.
by default and until this change, we assumed that the inverter
controlled by the DPL is powered by a battery. not all users have a
battery in their system. they still use the DPL to achieve net-zero
export. those users can now tell the DPL that their inverter is powered
by solar modules rather than a battery and the DPL will behave
accordingly.
do not scale limit if inverter is not producing, as DC channel power is
expected to be close to zero anyways.
do not scale limit if current inverter limit is small, such that
channels might produce very little power exactly because the limit is so
low.
move the calculation out of setNewPowerLimit and into a new function, so
that we can make use of return statements there.
the update frequency of Victron MPPT charger data, the battery Soc, the
huawei charger power, and the power meter differ from one another, and
differ in particular from the inverter update frequency.
the OnBattery-specific data is now handled in a new method, outside the
upstream code, which merely call the new function(s). the new function
will update the websocket independently from inverter updates. also, it
adds the respective data if it actually changed since it was last
updated through the websocket.
for the webapp to be able to recover in case of errors, all values are
also written to the websocket with a fixed interval of 10 seconds.
we previously only called commitPowerLimit() if the desired limit
changed such that the change was bigger than the hysteresis. we found
that if the limit update was not received and the desired limit would
not change much, the limit of the inverter was wrong for a long time.
to mitigate this, we introduced re-sending the limit update every 60
seconds, regardless of what the limit reported by the inverter was at
that time.
if the power-up command was not received, we also would repeat it only
once every 60 seconds.
this leads to a new kind of staleness and the actual inverter state was
still not matching the desired state.
this new approach effectively adds an additional control loop at the
start of the DPL loop(). that new function compares the requested
inverter state to the actual reported state. it sends updates (limit
update or power on state) until the desired inverter state is reached,
or until a (hard-coded) timeout occurs.
this approach also allows us to send power-up, power-down, and limit
update commands independent from one another and in a particular order.
this should make sure that the inverter is in the desired state even if
conditions change slowly and commands were not received as expected.
* make efficient use of available display area
fix calculation of the text baselines, using getAscent() in favor of
getMaxCharHeight(), which includes ascent and descent. this moves the
first text up and allows to insert margin between the lines until the
display area is fully utilized.
on large displays, if the small diagram is selected, keep the first line
rather low to avoid collision with the diagram y-axis label. in this mode,
there is still more space between the text lines as before, allowing for
improved readability.
* Feature: show power grid usage on display
if the power meter is enabled, the display will use two of three out
of every three-second time slot to show the grid consumption.
closes#620.
the DPL is interested in the battery's voltage to make decisions about
draining the battery or letting it charge (if the user opts to use
voltage thresholds rather than SoC thresholds). using the DC input
voltage reported by the inverter under control has disadvantages:
* the data might be quite old due to the communication protocol
implementation. more inverters being polled means even more lag. the
connection being wireless makes this even worse, due to the need
to retry the occasional lost packet, etc.
* the data is not very accurate, since the DC input of the inverter is
actually some cabling and a couple of junctions away from the actual
battery. this voltage drop can mostly only be estimated and is worse
with higher load. the load correction factor is there to mitigate
this, but it has its own problems and is cumbersome to calibrate.
instead, this change aims to use more accurate battery voltage readings,
if possible. the DPL now prefers the voltage as reported by the BMS,
since it is for sure the closest to the battery of all measuring points
and measures its voltage accurately regardless of the load (the voltage
reading will still drop with higher loads, but this will be only due to
the battery's internal resistance, not that of cabling or junctions). if
no BMS voltage reading is available, the DPL will instead use the charge
controller's voltage reading, as it is available with much higher
frequency and is assumed to be more accurate as it offers a resolution
of 10mV. only if none of these two sources can be used, the inverter DC
input voltage is assumed as the battery voltage.
closes#655.
the Victron SmartShunt communicates the SoC value in permille. this
should be displayed in the web UI accordingly. this is a good excuse to
fully move ownership of the SoC value to the BatteryStats base class and
add a precision indicator variable. this is required to be set each time
a derived class (a battery provider) wants to update the SoC value. the
precision is then used when populating the JSON data for the web UI
(live view).
related to #573.
in the respective context, the DPL only needs to be sure that the SoC
value is not outdated. it should not even care about other values
reported by the battery interface. hence, the isValid() method shall be
concerned with the SoC value timestamp only. the method is renamed for
clarity.
this extends the MqttBattery implementation by an additional topic which
allows to subscribe to receive battery voltage readings through the MQTT
broker. similar to the battery SoC topic, this allows to import a
critical battery data point for the DPL, in case the user chooses to use
voltage thresholds rather than SoC thresholds to control the DPL. if an
otherwise incompatible BMS is available which publishes the battery pack
voltage through MQTT, this can now be used to feed accurate voltage
readings to the DPL.
the BatteryStats base class shall be able to tell the total battery pack
voltage. for that reason, and to avoid code duplication, the voltage is
now handled in the base class and treated as a datum that is common to
all battery providers.
unfortunately, the battery SoC values reported by battery BMSs are
unreliable, at least for some users, or at least without regular
(manual) full charge cycles to calibrate the BMS. it offers great
advantages to connect OpenDTU-OnBattery to a BMS (MQTT publishing of
values, Home Assistent integration, etc.), but previously the users
were then forced to configure the DPL by SoC values.
this change allows to configure the DPL such that SoC values are
ignored. instead, the voltage limits are used to make DPL decisions, as
if no SoC was available in the first place.
the SoC related setting are hidden from the DPL settings view if SoC
values are configured to be ignored.
closes#654.
this implementation avoids the display content jumping the full
screensaver offset from right to left when the modulo operator
wraps. this change makes the display content walk from right to
left as it did walk from left to right.
in case the total production is larger than 1 MWh, i.e., 1000 kWh, the
text on the respective line becomes too large such that it reaches out
of the display when the screensaver is enabled.
this happens on the small and large displays.
this change switches the number format to a float without decimal places
if the total production is larger or equal to 1000 kWh. this saves a dot
and a digit, making the text short enough to fit the display even when
the screensaver moved the display contents as far to the right as it
does.
fix calculation of the text baselines, using getAscent() in favor of
getMaxCharHeight(), which includes ascent and descent. this moves the
first text up and allows to insert margin between the lines until the
display area is fully utilized.
on large displays, if the small diagram is selected, keep the first line
rather low to avoid collision with the diagram y-axis label. in this mode,
there is still more space between the text lines as before, allowing for
improved readability.
* pylontech HA integration: remove unused method/variable
* make MqttHandlePylontechHassClass::publishConfig() private.
there are no outside users of that method.
* rename to MqttHandleBatteryHass
* battery HA integration: merge methods and bring back forceUpdate().
even though the forceUpdate() method was not in use before, it makes
sense to implement it and use it when the battery config changes.
rather than controlling a separate flag, it now changes the _doPublish
flag of the class, which also triggers publishing the device config to
Home Assistant when an MQTT connection problem was detected. since
both situations are now handled similarly, we can merge the loop() and
publishConfig() methods.
* battery: provider specific sensors for HA
* move Battery MQTT loop to BatteryStats
the BatteryStats class should handle the MQTT publishing, including the
interval. for the calculation of a reasonable Home Assistent expiration
value this class now also knows the maximum publish interval.
* JK BMS: fix publishing values for Home Assistent
Home Assistent values expire, because we set them to expire after three
MQTT publish durations. for that reason, we need to re-publish all
values after our self-inflicted full publish interval.
* define JK BMS sensors for Home Assistent
closes#482.
previously, the Pytlontech Home Assistent class implementation had an
init() method, that was never called, as it did nothing. the class
relied on its loop() method being called from the main loop(). after
switching to the TaskScheduler approach, the Pylontech Home Assistent
class init() method was adjusted to register a task that calls the
loop() method periodically. however, the init() method was still not
called.
To reduce the heap usage it is necessary to send the inverters one by one instead of a huge response. A simple call to `/api/livedata/status` returns just some very general information. If detailed inverter information are required the inverter serial number has to appended `?inv=<serial number>`.
The websocket also returns only one inverter at a time. It as to be assembled at client side.
currently the whole SmartShunt data structure is copied to the
BatteryStats instance in every loop, even though the data cannot
possibly have changed. this is quite an expensive task to do in every
loop. this change tracks the last update timestamp and only does the
copy operation if an actual updated data structure was received from
the smart shunt.
this change adds the values of ESP.gteMaxAllocHeap() and
ESP.getMinFreeHead() to the prometheus metrics and the system
information object. the web UI uses these values to diplay the size of
the largest free contiguous block, calculate a rough estimate for the
level of fragmentation, and the maximum usage of heap memory since boot
in absolute and relative amounts.
this change adds the values of ESP.gteMaxAllocHeap() and
ESP.getMinFreeHead() to the prometheus metrics and the system
information object. the web UI uses these values to diplay the size of
the largest free contiguous block, calculate a rough estimate for the
level of fragmentation, and the maximum usage of heap memory since boot
in absolute and relative amounts.
OpenDTU console gets spammed with "WifiGeneric::hostByName() error when first trying to resolve the hostname via DNS. So reverse order: first try mDNS, if that fails try DNS. Also ensure that https bool is passed correctly to HTTPClient::begin(). Lastly, concatenate strings for building Digest authorization using "+" and not via snprintf.
